Apr 16 16:01:18.496746 ip-10-0-134-41 systemd[1]: Starting Kubernetes Kubelet...
Apr 16 16:01:18.893952 ip-10-0-134-41 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 16 16:01:18.893952 ip-10-0-134-41 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Apr 16 16:01:18.893952 ip-10-0-134-41 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 16 16:01:18.893952 ip-10-0-134-41 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Apr 16 16:01:18.893952 ip-10-0-134-41 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Apr 16 16:01:18.895395 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.895326    2578 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Apr 16 16:01:18.902600 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902574    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:01:18.902600 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902594    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:01:18.902600 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902598    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:01:18.902600 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902601    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:01:18.902600 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902604    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:01:18.902600 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902607    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:01:18.902600 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902610    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902612    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902615    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902618    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902621    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902624    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902626    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902629    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902631    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902634    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902637    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902639    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902642    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902645    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902647    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902650    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902654    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902658    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902661    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902665    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:01:18.902841 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902668    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902671    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902674    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902676    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902679    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902682    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902685    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902687    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902690    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902692    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902695    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902697    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902700    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902704    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902707    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902710    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902713    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902715    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902718    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902721    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:01:18.903342 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902723    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902726    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902728    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902731    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902733    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902736    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902738    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902741    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902743    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902746    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902748    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902752    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902756    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902758    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902761    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902763    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902766    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902768    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902770    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902773    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:01:18.903883 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902775    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902778    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902780    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902783    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902785    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902789    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902793    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902795    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902798    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902800    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902803    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902805    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902809    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902812    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902815    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902817    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902820    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902822    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902824    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:01:18.904353 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.902827    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903204    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903211    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903214    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903218    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903220    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903223    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903226    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903228    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903231    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903234    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903236    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903239    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903241    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903244    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903246    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903249    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903251    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903253    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:01:18.904832 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903256    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903260    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903262    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903265    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903267    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903270    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903273    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903276    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903278    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903281    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903283    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903285    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903288    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903290    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903293    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903296    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903298    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903301    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903303    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903305    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:01:18.905336 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903308    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903310    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903313    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903315    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903318    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903320    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903323    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903325    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903327    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903330    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903332    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903334    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903337    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903340    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903343    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903345    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903348    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903350    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903353    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903357    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:01:18.905890 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903360    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903364    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903367    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903370    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903373    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903375    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903378    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903380    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903383    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903385    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903387    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903390    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903392    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903395    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903397    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903399    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903402    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903404    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903407    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903409    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:01:18.906373 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903412    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903414    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903417    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903420    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903424    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903428    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903431    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.903434    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904174    2578 flags.go:64] FLAG: --address="0.0.0.0"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904182    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904189    2578 flags.go:64] FLAG: --anonymous-auth="true"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904193    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904198    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904201    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904205    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904209    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904212    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904215    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904219    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904222    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904225    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904227    2578 flags.go:64] FLAG: --cgroup-root=""
Apr 16 16:01:18.906885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904230    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904233    2578 flags.go:64] FLAG: --client-ca-file=""
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904236    2578 flags.go:64] FLAG: --cloud-config=""
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904239    2578 flags.go:64] FLAG: --cloud-provider="external"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904242    2578 flags.go:64] FLAG: --cluster-dns="[]"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904246    2578 flags.go:64] FLAG: --cluster-domain=""
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904249    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904252    2578 flags.go:64] FLAG: --config-dir=""
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904255    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904258    2578 flags.go:64] FLAG: --container-log-max-files="5"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904262    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904264    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904267    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904271    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904274    2578 flags.go:64] FLAG: --contention-profiling="false"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904277    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904279    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904282    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904285    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904289    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904292    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904294    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904297    2578 flags.go:64] FLAG: --enable-load-reader="false"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904300    2578 flags.go:64] FLAG: --enable-server="true"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904303    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 16 16:01:18.907421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904308    2578 flags.go:64] FLAG: --event-burst="100"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904311    2578 flags.go:64] FLAG: --event-qps="50"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904313    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904316    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904319    2578 flags.go:64] FLAG: --eviction-hard=""
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904323    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904326    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904329    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904332    2578 flags.go:64] FLAG: --eviction-soft=""
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904335    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904338    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904340    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904343    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904346    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904349    2578 flags.go:64] FLAG: --fail-swap-on="true"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904351    2578 flags.go:64] FLAG: --feature-gates=""
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904355    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904358    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904361    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904365    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904368    2578 flags.go:64] FLAG: --healthz-port="10248"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904371    2578 flags.go:64] FLAG: --help="false"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904374    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-41.ec2.internal"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904377    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 16 16:01:18.908035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904380    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904383    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904387    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904390    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904393    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904396    2578 flags.go:64] FLAG: --image-service-endpoint=""
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904398    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904402    2578 flags.go:64] FLAG: --kube-api-burst="100"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904404    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904407    2578 flags.go:64] FLAG: --kube-api-qps="50"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904410    2578 flags.go:64] FLAG: --kube-reserved=""
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904413    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904415    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904418    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904421    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904424    2578 flags.go:64] FLAG: --lock-file=""
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904427    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904429    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904432    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904437    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904462    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904465    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904468    2578 flags.go:64] FLAG: --logging-format="text"
Apr 16 16:01:18.908637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904470    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904474    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904477    2578 flags.go:64] FLAG: --manifest-url=""
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904480    2578 flags.go:64] FLAG: --manifest-url-header=""
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904485    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904488    2578 flags.go:64] FLAG: --max-open-files="1000000"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904492    2578 flags.go:64] FLAG: --max-pods="110"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904495    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904498    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904500    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904503    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904506    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904509    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904512    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904518    2578 flags.go:64] FLAG: --node-status-max-images="50"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904521    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904526    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904530    2578 flags.go:64] FLAG: --pod-cidr=""
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904533    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc76bab72f320de3d4105c90d73c4fb139c09e20ce0fa8dcbc0cb59920d27dec"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904538    2578 flags.go:64] FLAG: --pod-manifest-path=""
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904541    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904544    2578 flags.go:64] FLAG: --pods-per-core="0"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904547    2578 flags.go:64] FLAG: --port="10250"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904550    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 16 16:01:18.909202 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904553    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09cb9f2d732f3eee0"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904556    2578 flags.go:64] FLAG: --qos-reserved=""
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904559    2578 flags.go:64] FLAG: --read-only-port="10255"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904561    2578 flags.go:64] FLAG: --register-node="true"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904564    2578 flags.go:64] FLAG: --register-schedulable="true"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904567    2578 flags.go:64] FLAG: --register-with-taints=""
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904571    2578 flags.go:64] FLAG: --registry-burst="10"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904573    2578 flags.go:64] FLAG: --registry-qps="5"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904576    2578 flags.go:64] FLAG: --reserved-cpus=""
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904578    2578 flags.go:64] FLAG: --reserved-memory=""
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904582    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904585    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904588    2578 flags.go:64] FLAG: --rotate-certificates="false"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904590    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904593    2578 flags.go:64] FLAG: --runonce="false"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904596    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904598    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904601    2578 flags.go:64] FLAG: --seccomp-default="false"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904604    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904606    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904609    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904612    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904615    2578 flags.go:64] FLAG: --storage-driver-password="root"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904617    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904621    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904624    2578 flags.go:64] FLAG: --storage-driver-user="root"
Apr 16 16:01:18.909792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904627    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904630    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904633    2578 flags.go:64] FLAG: --system-cgroups=""
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904635    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904640    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904643    2578 flags.go:64] FLAG: --tls-cert-file=""
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904646    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904650    2578 flags.go:64] FLAG: --tls-min-version=""
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904652    2578 flags.go:64] FLAG: --tls-private-key-file=""
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904655    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904657    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904660    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904663    2578 flags.go:64] FLAG: --v="2"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904667    2578 flags.go:64] FLAG: --version="false"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904671    2578 flags.go:64] FLAG: --vmodule=""
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904675    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.904678    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904767    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904771    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904774    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904777    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904780    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904783    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:01:18.910407 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904787    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904792    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904795    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904797    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904800    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904803    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904805    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904812    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904814    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904817    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904819    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904822    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904825    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904827    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904830    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904832    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904834    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904837    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904840    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904842    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:01:18.910971 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904845    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904848    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904851    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904853    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904856    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904858    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904861    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904864    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904866    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904869    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904871    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904874    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904876    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904879    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904881    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904883    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904886    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904888    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904890    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904894    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:01:18.911493 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904896    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904898    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904901    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904904    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904906    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904908    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904911    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904913    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904916    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904918    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904921    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904923    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904926    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904928    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904931    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904934    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904936    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904939    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904941    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904944    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:01:18.911985 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904946    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904948    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904951    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904953    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904955    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904958    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904961    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904963    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904965    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904968    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904972    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904975    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904978    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904981    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904984    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904987    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904990    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904993    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904995    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:01:18.912484 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.904998    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.905853    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.912748    2578 server.go:530] "Kubelet version" kubeletVersion="v1.33.9"
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.912762    2578 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912807    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912811    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912816    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912820    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912824    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912827    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912830    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912833    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912836    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912839    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912841    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:01:18.912960 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912844    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912847    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912849    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912852    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912854    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912857    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912860    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912862    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912865    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912867    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912869    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912872    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912874    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912877    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912879    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912882    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912884    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912887    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912889    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912892    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:01:18.913352 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912896    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912898    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912901    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912903    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912906    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912908    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912910    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912913    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912916    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912918    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912920    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912923    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912925    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912927    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912930    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912932    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912935    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912937    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912940    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912942    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:01:18.913889 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912944    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912947    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912949    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912952    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912955    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912957    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912960    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912962    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912965    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912967    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912970    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912973    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912976    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912979    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912982    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912986    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912988    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912991    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912993    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:01:18.914377 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912996    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.912998    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913001    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913003    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913006    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913008    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913011    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913013    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913016    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913018    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913020    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913023    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913025    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913028    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913030    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:01:18.914979 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913032    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.913038    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913139    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913144    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913147    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913150    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913153    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913156    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913158    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913161    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913164    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913167    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913170    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913173    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913176    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913178    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Apr 16 16:01:18.915346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913180    2578 feature_gate.go:328] unrecognized feature gate: Example
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913184    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913187    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913190    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913192    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913195    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913205    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913207    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913210    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913212    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913215    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913217    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913219    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913222    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913224    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913227    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913229    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913232    2578 feature_gate.go:328] unrecognized feature gate: Example2
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913234    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913236    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Apr 16 16:01:18.915754 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913239    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913241    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913244    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913246    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913249    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913251    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913254    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913257    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913259    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913262    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913265    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913267    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913269    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913272    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913274    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913277    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913279    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913281    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913284    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913286    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Apr 16 16:01:18.916239 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913289    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913291    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913293    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913296    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913298    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913300    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913303    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913305    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913308    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913310    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913313    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913315    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913317    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913320    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913322    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913325    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913327    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913330    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913333    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913335    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Apr 16 16:01:18.916735 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913338    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913340    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913343    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913346    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913348    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913350    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913354    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913358    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913360    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913364    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913366    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:18.913369    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.913373    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Apr 16 16:01:18.917219 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.914097    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Apr 16 16:01:18.918216 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.918202    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Apr 16 16:01:18.919237 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.919226    2578 server.go:1019] "Starting client certificate rotation"
Apr 16 16:01:18.919338 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.919323    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 16 16:01:18.919376 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.919361    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Apr 16 16:01:18.942161 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.942144    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 16 16:01:18.946491 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.946472    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Apr 16 16:01:18.959436 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.959420    2578 log.go:25] "Validated CRI v1 runtime API"
Apr 16 16:01:18.964073 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.964055    2578 log.go:25] "Validated CRI v1 image API"
Apr 16 16:01:18.966629 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.966615    2578 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Apr 16 16:01:18.969438 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.969417    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ac75d5c8-f566-424b-903d-4b00821b6417:/dev/nvme0n1p3 d03c5fde-caa7-415a-b241-3890d5babc4c:/dev/nvme0n1p4]
Apr 16 16:01:18.969543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.969436    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Apr 16 16:01:18.971302 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.971282    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 16 16:01:18.976392 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.976280    2578 manager.go:217] Machine: {Timestamp:2026-04-16 16:01:18.974415564 +0000 UTC m=+0.370499975 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124775 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec262bdaa68d4112ea8bc63049ca906d SystemUUID:ec262bda-a68d-4112-ea8b-c63049ca906d BootID:63ffebf6-554f-4e7a-8dde-8c004cf41eb3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6098944 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:55:d4:5d:1b:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:55:d4:5d:1b:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:25:dc:00:70:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 16 16:01:18.976392 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.976381    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 16 16:01:18.976577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.976494    2578 manager.go:233] Version: {KernelVersion:5.14.0-570.104.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260401-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 16 16:01:18.978081 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.978057    2578 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 16 16:01:18.978215 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.978084    2578 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-41.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Apr 16 16:01:18.978264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.978224    2578 topology_manager.go:138] "Creating topology manager with none policy"
Apr 16 16:01:18.978264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.978233    2578 container_manager_linux.go:306] "Creating device plugin manager"
Apr 16 16:01:18.978264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.978245    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 16 16:01:18.978938 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.978927    2578 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 16 16:01:18.979830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.979821    2578 state_mem.go:36] "Initialized new in-memory state store"
Apr 16 16:01:18.979929 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.979920    2578 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Apr 16 16:01:18.981915 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.981906    2578 kubelet.go:491] "Attempting to sync node with API server"
Apr 16 16:01:18.981949 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.981919    2578 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 16 16:01:18.981949 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.981930    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Apr 16 16:01:18.981949 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.981939    2578 kubelet.go:397] "Adding apiserver pod source"
Apr 16 16:01:18.981949 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.981947    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 16 16:01:18.983052 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.983036    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 16 16:01:18.983137 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.983057    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Apr 16 16:01:18.985998 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.985982    2578 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1"
Apr 16 16:01:18.987772 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.987758    2578 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Apr 16 16:01:18.989064 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989050    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989073    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989083    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989092    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989098    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989103    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989110    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989116    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Apr 16 16:01:18.989122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989122    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Apr 16 16:01:18.989330 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989128    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Apr 16 16:01:18.989330 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989141    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Apr 16 16:01:18.989330 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.989150    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Apr 16 16:01:18.990773 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.990761    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Apr 16 16:01:18.990773 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.990774    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Apr 16 16:01:18.994107 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.994091    2578 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Apr 16 16:01:18.994187 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.994134    2578 server.go:1295] "Started kubelet"
Apr 16 16:01:18.994264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.994228    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Apr 16 16:01:18.995482 ip-10-0-134-41 systemd[1]: Started Kubernetes Kubelet.
Apr 16 16:01:18.996166 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.994222    2578 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Apr 16 16:01:18.996519 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.996384    2578 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-41.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Apr 16 16:01:18.996990 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:18.996962    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Apr 16 16:01:18.997185 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:18.997160    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Apr 16 16:01:18.997273 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.997248    2578 server_v1.go:47] "podresources" method="list" useActivePods=true
Apr 16 16:01:18.997955 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:18.997811    2578 server.go:317] "Adding debug handlers to kubelet server"
Apr 16 16:01:19.001635 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.001617    2578 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Apr 16 16:01:19.003718 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.003688    2578 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Apr 16 16:01:19.004055 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.003177    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-41.ec2.internal.18a6e1b6f7556182  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-41.ec2.internal,UID:ip-10-0-134-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-41.ec2.internal,},FirstTimestamp:2026-04-16 16:01:18.994104706 +0000 UTC m=+0.390189116,LastTimestamp:2026-04-16 16:01:18.994104706 +0000 UTC m=+0.390189116,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-41.ec2.internal,}"
Apr 16 16:01:19.005140 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.005124    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Apr 16 16:01:19.005184 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.005137    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Apr 16 16:01:19.005775 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.005758    2578 factory.go:55] Registering systemd factory
Apr 16 16:01:19.005775 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.005776    2578 factory.go:223] Registration of the systemd container factory successfully
Apr 16 16:01:19.005900 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.005872    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.005964 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.005897    2578 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Apr 16 16:01:19.006024 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.005900    2578 volume_manager.go:295] "The desired_state_of_world populator starts"
Apr 16 16:01:19.006069 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006047    2578 factory.go:153] Registering CRI-O factory
Apr 16 16:01:19.006115 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006073    2578 factory.go:223] Registration of the crio container factory successfully
Apr 16 16:01:19.006115 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006076    2578 volume_manager.go:297] "Starting Kubelet Volume Manager"
Apr 16 16:01:19.006208 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006137    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Apr 16 16:01:19.006208 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006156    2578 factory.go:103] Registering Raw factory
Apr 16 16:01:19.006208 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006172    2578 manager.go:1196] Started watching for new ooms in manager
Apr 16 16:01:19.006337 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006216    2578 reconstruct.go:97] "Volume reconstruction finished"
Apr 16 16:01:19.006337 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006226    2578 reconciler.go:26] "Reconciler: start to sync state"
Apr 16 16:01:19.006901 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.006615    2578 manager.go:319] Starting recovery of all containers
Apr 16 16:01:19.010033 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.010005    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-41.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Apr 16 16:01:19.010124 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.010077    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Apr 16 16:01:19.016597 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.016464    2578 manager.go:324] Recovery completed
Apr 16 16:01:19.018245 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.018224    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Apr 16 16:01:19.020744 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.020728    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bf8jz"
Apr 16 16:01:19.021078 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.021065    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:01:19.023677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.023662    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:01:19.023755 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.023688    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:01:19.023755 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.023702    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:01:19.024156 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.024142    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Apr 16 16:01:19.024196 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.024166    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Apr 16 16:01:19.024196 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.024180    2578 state_mem.go:36] "Initialized new in-memory state store"
Apr 16 16:01:19.025644 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.025586    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-41.ec2.internal.18a6e1b6f91896f4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-41.ec2.internal,UID:ip-10-0-134-41.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-41.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-41.ec2.internal,},FirstTimestamp:2026-04-16 16:01:19.023675124 +0000 UTC m=+0.419759534,LastTimestamp:2026-04-16 16:01:19.023675124 +0000 UTC m=+0.419759534,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-41.ec2.internal,}"
Apr 16 16:01:19.027016 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.027003    2578 policy_none.go:49] "None policy: Start"
Apr 16 16:01:19.027062 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.027022    2578 memory_manager.go:186] "Starting memorymanager" policy="None"
Apr 16 16:01:19.027062 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.027032    2578 state_mem.go:35] "Initializing new in-memory state store"
Apr 16 16:01:19.029576 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.029563    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bf8jz"
Apr 16 16:01:19.068000 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.067986    2578 manager.go:341] "Starting Device Plugin manager"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.068032    2578 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.068044    2578 server.go:85] "Starting device plugin registration server"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.068255    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.068266    2578 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.068386    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.068523    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.068532    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.069011    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Apr 16 16:01:19.075212 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.069048    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.101091 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.101066    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Apr 16 16:01:19.102311 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.102298    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Apr 16 16:01:19.102362 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.102321    2578 status_manager.go:230] "Starting to sync pod status with apiserver"
Apr 16 16:01:19.102362 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.102335    2578 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Apr 16 16:01:19.102362 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.102341    2578 kubelet.go:2451] "Starting kubelet main sync loop"
Apr 16 16:01:19.102476 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.102369    2578 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Apr 16 16:01:19.106664 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.106643    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:01:19.168819 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.168772    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:01:19.169560 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.169544    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:01:19.169637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.169573    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:01:19.169637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.169582    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:01:19.169637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.169602    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.181487 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.181471    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.181487 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.181488    2578 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-41.ec2.internal\": node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.203350 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.203334    2578 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"]
Apr 16 16:01:19.203436 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.203386    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:01:19.205313 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.205300    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:01:19.205368 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.205319    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:01:19.205368 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.205328    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:01:19.206587 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.206576    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:01:19.206735 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.206720    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.206771 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.206749    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:01:19.207215 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.207197    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:01:19.207304 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.207229    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:01:19.207304 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.207243    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:01:19.207304 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.207228    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:01:19.207397 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.207320    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:01:19.207397 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.207334    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:01:19.209064 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.209048    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.209127 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.209083    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Apr 16 16:01:19.209802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.209787    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientMemory"
Apr 16 16:01:19.209861 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.209819    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasNoDiskPressure"
Apr 16 16:01:19.209861 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.209830    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeHasSufficientPID"
Apr 16 16:01:19.216530 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.216517    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.224934 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.224918    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-41.ec2.internal\" not found" node="ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.228817 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.228800    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-41.ec2.internal\" not found" node="ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.306998 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.306977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2d68b261ea36bffca5aca5d127db206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"e2d68b261ea36bffca5aca5d127db206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.307062 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.307003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d68b261ea36bffca5aca5d127db206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"e2d68b261ea36bffca5aca5d127db206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.316995 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.316976    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.407295 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.407277    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2d68b261ea36bffca5aca5d127db206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"e2d68b261ea36bffca5aca5d127db206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.407360 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.407303    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d68b261ea36bffca5aca5d127db206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"e2d68b261ea36bffca5aca5d127db206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.407407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.407352    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2d68b261ea36bffca5aca5d127db206-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"e2d68b261ea36bffca5aca5d127db206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.407407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.407382    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2d68b261ea36bffca5aca5d127db206-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal\" (UID: \"e2d68b261ea36bffca5aca5d127db206\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.407407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.407396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac3c29e3e9ef322bb0cc9eb018b42ffc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-41.ec2.internal\" (UID: \"ac3c29e3e9ef322bb0cc9eb018b42ffc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.417395 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.417378    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.508020 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.507963    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac3c29e3e9ef322bb0cc9eb018b42ffc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-41.ec2.internal\" (UID: \"ac3c29e3e9ef322bb0cc9eb018b42ffc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.508020 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.508013    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ac3c29e3e9ef322bb0cc9eb018b42ffc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-41.ec2.internal\" (UID: \"ac3c29e3e9ef322bb0cc9eb018b42ffc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.518104 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.518083    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.529259 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.529239    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.531976 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.531963    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:19.618566 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.618538    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.719101 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.719081    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.819609 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.819542    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:19.919262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.919240    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Apr 16 16:01:19.919654 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:19.919354    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Apr 16 16:01:19.920317 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:19.920302    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:20.005430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.005399    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Apr 16 16:01:20.016077 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.016056    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Apr 16 16:01:20.020374 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:20.020357    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:20.022040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.022024    2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:01:20.030900 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.030874    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-15 15:56:19 +0000 UTC" deadline="2028-01-04 06:29:21.026313122 +0000 UTC"
Apr 16 16:01:20.030900 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.030897    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15062h28m0.995418654s"
Apr 16 16:01:20.037335 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.037299    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjtfb"
Apr 16 16:01:20.043507 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.043493    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kjtfb"
Apr 16 16:01:20.121273 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:20.121227    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-41.ec2.internal\" not found"
Apr 16 16:01:20.126652 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.126634    2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:01:20.137191 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:20.137163    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3c29e3e9ef322bb0cc9eb018b42ffc.slice/crio-99a1ca0851d710ae2d7d07a01e8a65bb7aa0cfb9a2f472cf724d891ef35fee5a WatchSource:0}: Error finding container 99a1ca0851d710ae2d7d07a01e8a65bb7aa0cfb9a2f472cf724d891ef35fee5a: Status 404 returned error can't find the container with id 99a1ca0851d710ae2d7d07a01e8a65bb7aa0cfb9a2f472cf724d891ef35fee5a
Apr 16 16:01:20.137685 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:20.137665    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d68b261ea36bffca5aca5d127db206.slice/crio-434baa6765b5b8f3581486e06de798b40e2fc1f823f35760385a41f7fa1189f4 WatchSource:0}: Error finding container 434baa6765b5b8f3581486e06de798b40e2fc1f823f35760385a41f7fa1189f4: Status 404 returned error can't find the container with id 434baa6765b5b8f3581486e06de798b40e2fc1f823f35760385a41f7fa1189f4
Apr 16 16:01:20.141358 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.141346    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:01:20.179842 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.179820    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:01:20.205845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.205824    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:20.218744 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.218723    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 16 16:01:20.219597 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.219574    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal"
Apr 16 16:01:20.226009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.225993    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Apr 16 16:01:20.723889 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.723718    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Apr 16 16:01:20.983549 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.983442    2578 apiserver.go:52] "Watching apiserver"
Apr 16 16:01:20.990054 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.990031    2578 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Apr 16 16:01:20.991180 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.991156    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-nwd2d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal","openshift-multus/multus-additional-cni-plugins-52kws","openshift-multus/multus-xkfxp","kube-system/konnectivity-agent-j5rnw","openshift-dns/node-resolver-7c442","openshift-multus/network-metrics-daemon-pbf54","openshift-network-diagnostics/network-check-target-8849t","openshift-network-operator/iptables-alerter-rxkkz","openshift-ovn-kubernetes/ovnkube-node-9n7wn","kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj","openshift-cluster-node-tuning-operator/tuned-p6drl"]
Apr 16 16:01:20.993398 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.993377    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:20.993525 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:20.993483    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:20.994598 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.994575    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:20.995801 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.995779    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:20.997221 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.997199    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:20.997408 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.997384    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Apr 16 16:01:20.997645 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.997628    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Apr 16 16:01:20.998183 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.998166    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Apr 16 16:01:20.998281 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.998266    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9tznm\""
Apr 16 16:01:20.998535 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.998512    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Apr 16 16:01:20.998729 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.998713    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Apr 16 16:01:20.998904 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.998887    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Apr 16 16:01:20.999070 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.999057    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q22dv\""
Apr 16 16:01:20.999437 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.999423    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Apr 16 16:01:20.999775 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:20.999758    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-66kbc\""
Apr 16 16:01:21.000084 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.000066    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Apr 16 16:01:21.000973 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.000955    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.001123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.001102    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:21.001196 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.001166    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:21.002582 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.002565    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.002676 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.002640    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.003206 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.003191    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Apr 16 16:01:21.003713 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.003438    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sxb74\""
Apr 16 16:01:21.003713 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.003627    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Apr 16 16:01:21.004254 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.004237    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.005418 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.005399    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Apr 16 16:01:21.005551 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.005534    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Apr 16 16:01:21.005922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.005623    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7xt72\""
Apr 16 16:01:21.005922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.005639    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5hqq4\""
Apr 16 16:01:21.005922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.005720    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Apr 16 16:01:21.005922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.005749    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Apr 16 16:01:21.005922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.005821    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Apr 16 16:01:21.006203 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.006057    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.008095 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.008071    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Apr 16 16:01:21.008307 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.008292    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Apr 16 16:01:21.008535 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.008517    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Apr 16 16:01:21.008771 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.008756    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Apr 16 16:01:21.009020 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.009003    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Apr 16 16:01:21.009395 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.009376    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Apr 16 16:01:21.009737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.009718    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9d7ct\""
Apr 16 16:01:21.010641 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.010623    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Apr 16 16:01:21.011175 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.011155    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Apr 16 16:01:21.011502 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.011481    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Apr 16 16:01:21.011885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.011867    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Apr 16 16:01:21.012276 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.012259    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9lfkb\""
Apr 16 16:01:21.013128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.013106    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.015565 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.015544    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Apr 16 16:01:21.015792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.015775    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5wrm7\""
Apr 16 16:01:21.016020 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016001    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Apr 16 16:01:21.016115 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-multus-certs\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.016115 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndljn\" (UniqueName: \"kubernetes.io/projected/169cfa67-f1fa-4803-84bc-2219a6384156-kube-api-access-ndljn\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.016115 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-system-cni-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.016115 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26fd2187-a513-457a-9763-91f0fdff4ad6-host\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.016308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-run-netns\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.016308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.016308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysctl-d\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.016308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-cni-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.016308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-cnibin\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.016308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-etc-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.016308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-run\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-slash\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-ovn\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-registration-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kv7j\" (UniqueName: \"kubernetes.io/projected/547d3bd9-3b02-47a3-a133-480d22876e5f-kube-api-access-8kv7j\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343a0f51-524e-4bf4-82df-e0253871139f-host-slash\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-modprobe-d\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-tuned\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2st\" (UniqueName: \"kubernetes.io/projected/26fd2187-a513-457a-9763-91f0fdff4ad6-kube-api-access-kh2st\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-ovnkube-config\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.016618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-var-lib-kubelet\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjk6s\" (UniqueName: \"kubernetes.io/projected/c7c9806f-cdc6-417b-95e3-0749a649dcef-kube-api-access-fjk6s\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-system-cni-dir\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysctl-conf\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-os-release\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-systemd\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-cni-binary-copy\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-hostroot\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26fd2187-a513-457a-9763-91f0fdff4ad6-serviceca\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-log-socket\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-socket-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.016957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ae8e22-0b95-4a1f-9939-147624118297-agent-certs\") pod \"konnectivity-agent-j5rnw\" (UID: \"05ae8e22-0b95-4a1f-9939-147624118297\") " pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-systemd\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-cni-bin\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.017147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-daemon-config\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-systemd-units\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-k8s-cni-cncf-io\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2j87\" (UniqueName: \"kubernetes.io/projected/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-kube-api-access-r2j87\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-node-log\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-lib-modules\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-host\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xdj\" (UniqueName: \"kubernetes.io/projected/5343bde2-108d-43c0-beaf-40395b901cce-kube-api-access-b9xdj\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-cni-multus\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-ovnkube-script-lib\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-device-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-kubelet\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wt2p\" (UniqueName: \"kubernetes.io/projected/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-kube-api-access-5wt2p\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlpk\" (UniqueName: \"kubernetes.io/projected/343a0f51-524e-4bf4-82df-e0253871139f-kube-api-access-5mlpk\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.018262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysconfig\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-cni-bin\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-cni-netd\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-netns\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d945238-4c81-4dc5-bf64-145e503a3eb3-hosts-file\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d945238-4c81-4dc5-bf64-145e503a3eb3-tmp-dir\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017748    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-var-lib-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ae8e22-0b95-4a1f-9939-147624118297-konnectivity-ca\") pod \"konnectivity-agent-j5rnw\" (UID: \"05ae8e22-0b95-4a1f-9939-147624118297\") " pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/343a0f51-524e-4bf4-82df-e0253871139f-iptables-alerter-script\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-os-release\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-kubelet\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4sbk\" (UniqueName: \"kubernetes.io/projected/2d945238-4c81-4dc5-bf64-145e503a3eb3-kube-api-access-t4sbk\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.017978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/169cfa67-f1fa-4803-84bc-2219a6384156-ovn-node-metrics-cert\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.019029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7c9806f-cdc6-417b-95e3-0749a649dcef-tmp\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-etc-kubernetes\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-env-overrides\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-sys-fs\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-cni-binary-copy\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-conf-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-kubernetes\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-sys\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-cnibin\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.019776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.018251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-socket-dir-parent\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.044621 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.044592    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 15:56:20 +0000 UTC" deadline="2028-01-11 21:53:36.41165933 +0000 UTC"
Apr 16 16:01:21.044621 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.044622    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15245h52m15.367041116s"
Apr 16 16:01:21.106942 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.106900    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" event={"ID":"ac3c29e3e9ef322bb0cc9eb018b42ffc","Type":"ContainerStarted","Data":"99a1ca0851d710ae2d7d07a01e8a65bb7aa0cfb9a2f472cf724d891ef35fee5a"}
Apr 16 16:01:21.107040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.106961    2578 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Apr 16 16:01:21.108079 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.108053    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" event={"ID":"e2d68b261ea36bffca5aca5d127db206","Type":"ContainerStarted","Data":"434baa6765b5b8f3581486e06de798b40e2fc1f823f35760385a41f7fa1189f4"}
Apr 16 16:01:21.119338 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119313    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r2j87\" (UniqueName: \"kubernetes.io/projected/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-kube-api-access-r2j87\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.119440 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119353    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-node-log\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119440 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119377    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-lib-modules\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.119440 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119414    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-host\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.119577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119438    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xdj\" (UniqueName: \"kubernetes.io/projected/5343bde2-108d-43c0-beaf-40395b901cce-kube-api-access-b9xdj\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.119577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119497    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-cni-multus\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.119577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119521    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119562    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-ovnkube-script-lib\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119585    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-device-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.119705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119620    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.119705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119645    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-kubelet\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119691    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5wt2p\" (UniqueName: \"kubernetes.io/projected/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-kube-api-access-5wt2p\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:21.119827 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119720    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlpk\" (UniqueName: \"kubernetes.io/projected/343a0f51-524e-4bf4-82df-e0253871139f-kube-api-access-5mlpk\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.119827 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119749    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysconfig\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.119827 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119776    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-cni-bin\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119827 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119795    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-cni-netd\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119827 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119808    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-netns\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.119974 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119853    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-node-log\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119974 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119855    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-netns\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.119974 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119897    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-device-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.119974 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119903    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.119974 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.119950    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-kubelet\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysconfig\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.120212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120179    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-lib-modules\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.120212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120200    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d945238-4c81-4dc5-bf64-145e503a3eb3-hosts-file\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.120412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120229    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-cni-netd\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120238    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d945238-4c81-4dc5-bf64-145e503a3eb3-tmp-dir\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.120412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120251    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-host\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.120412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120265    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-cni-multus\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.120412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120268    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-var-lib-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120311    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-var-lib-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120384    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120420    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ae8e22-0b95-4a1f-9939-147624118297-konnectivity-ca\") pod \"konnectivity-agent-j5rnw\" (UID: \"05ae8e22-0b95-4a1f-9939-147624118297\") " pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120441    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2d945238-4c81-4dc5-bf64-145e503a3eb3-hosts-file\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120481    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/343a0f51-524e-4bf4-82df-e0253871139f-iptables-alerter-script\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120544    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120574    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-os-release\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.120596    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120612    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-kubelet\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d945238-4c81-4dc5-bf64-145e503a3eb3-tmp-dir\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120654    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t4sbk\" (UniqueName: \"kubernetes.io/projected/2d945238-4c81-4dc5-bf64-145e503a3eb3-kube-api-access-t4sbk\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-cni-bin\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.120715    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs podName:53da8c88-a72c-4cf8-abde-f3a449cbdf3f nodeName:}" failed. No retries permitted until 2026-04-16 16:01:21.620645943 +0000 UTC m=+3.016730343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs") pod "network-metrics-daemon-pbf54" (UID: "53da8c88-a72c-4cf8-abde-f3a449cbdf3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120764    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120793    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/169cfa67-f1fa-4803-84bc-2219a6384156-ovn-node-metrics-cert\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120808    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7c9806f-cdc6-417b-95e3-0749a649dcef-tmp\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120825    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-etc-kubernetes\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120841    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-env-overrides\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.120855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120855    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-sys-fs\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120872    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-cni-binary-copy\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120888    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-conf-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120903    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-kubernetes\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120920    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-sys\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120936    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-cnibin\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120943    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-etc-kubernetes\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120953    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-socket-dir-parent\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-socket-dir-parent\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.120992    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-multus-certs\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121021    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ndljn\" (UniqueName: \"kubernetes.io/projected/169cfa67-f1fa-4803-84bc-2219a6384156-kube-api-access-ndljn\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121040    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-system-cni-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121057    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26fd2187-a513-457a-9763-91f0fdff4ad6-host\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-multus-certs\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121073    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-run-netns\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-sys-fs\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.121653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-run-ovn-kubernetes\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121108    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-ovnkube-script-lib\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121338    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121589    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ae8e22-0b95-4a1f-9939-147624118297-konnectivity-ca\") pod \"konnectivity-agent-j5rnw\" (UID: \"05ae8e22-0b95-4a1f-9939-147624118297\") " pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-run-netns\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121622    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-cni-binary-copy\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121692    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-kubernetes\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-os-release\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121760    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/343a0f51-524e-4bf4-82df-e0253871139f-iptables-alerter-script\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121767    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5343bde2-108d-43c0-beaf-40395b901cce-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121811    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121863    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysctl-d\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121897    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-system-cni-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121892    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-sys\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26fd2187-a513-457a-9763-91f0fdff4ad6-host\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121898    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-cni-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121927    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121993    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-cnibin\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.122356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122042    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-kubelet\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122051    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-cni-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.121973    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-cnibin\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122067    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-cnibin\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122062    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysctl-d\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122111    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-conf-dir\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122139    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-etc-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122178    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-run\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122239    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-etc-openvswitch\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122231    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-slash\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122266    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-env-overrides\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122288    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-run\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122324    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-ovn\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122345    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-slash\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122354    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-ovn\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122384    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-registration-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122436    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-registration-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.123299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122460    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8kv7j\" (UniqueName: \"kubernetes.io/projected/547d3bd9-3b02-47a3-a133-480d22876e5f-kube-api-access-8kv7j\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122496    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343a0f51-524e-4bf4-82df-e0253871139f-host-slash\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122518    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-modprobe-d\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122539    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-tuned\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/343a0f51-524e-4bf4-82df-e0253871139f-host-slash\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122583    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2st\" (UniqueName: \"kubernetes.io/projected/26fd2187-a513-457a-9763-91f0fdff4ad6-kube-api-access-kh2st\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122632    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122663    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-ovnkube-config\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.124138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.122469    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.125518 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125062    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-var-lib-kubelet\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.125613 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125544    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fjk6s\" (UniqueName: \"kubernetes.io/projected/c7c9806f-cdc6-417b-95e3-0749a649dcef-kube-api-access-fjk6s\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.125613 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125601    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-system-cni-dir\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.125726 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125653    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysctl-conf\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.125726 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125710    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-os-release\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.125817 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125735    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-systemd\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.125817 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125778    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.125913 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125842    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-cni-binary-copy\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.125913 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125873    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-hostroot\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.125913 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125904    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26fd2187-a513-457a-9763-91f0fdff4ad6-serviceca\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.126045 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125928    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-log-socket\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.126045 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125971    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.126156 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.126058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.126156 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.125473    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-modprobe-d\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.126582 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.126435    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-system-cni-dir\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.126669 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.126601    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-sysctl-conf\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.126732 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.126683    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-os-release\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.126780 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.126738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-run-systemd\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.127026 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.127007    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/169cfa67-f1fa-4803-84bc-2219a6384156-ovn-node-metrics-cert\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.127243 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.127224    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7c9806f-cdc6-417b-95e3-0749a649dcef-tmp\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.129710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.127507    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5343bde2-108d-43c0-beaf-40395b901cce-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.129710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.127787    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/169cfa67-f1fa-4803-84bc-2219a6384156-ovnkube-config\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.129710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.128205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-cni-binary-copy\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.129710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.128507    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-hostroot\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.129710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.128934    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/26fd2187-a513-457a-9763-91f0fdff4ad6-serviceca\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.129710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.129058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-log-socket\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.131464 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.130772    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlpk\" (UniqueName: \"kubernetes.io/projected/343a0f51-524e-4bf4-82df-e0253871139f-kube-api-access-5mlpk\") pod \"iptables-alerter-rxkkz\" (UID: \"343a0f51-524e-4bf4-82df-e0253871139f\") " pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.132093 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.132073    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-tuned\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.132623 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.132600    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-var-lib-kubelet\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.132873 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.132855    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:01:21.132946 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.132880    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:01:21.132946 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.132894    2578 projected.go:194] Error preparing data for projected volume kube-api-access-29nkz for pod openshift-network-diagnostics/network-check-target-8849t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:21.133047 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.132965    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz podName:342d6e09-3c3b-470d-b383-b470e1c3a086 nodeName:}" failed. No retries permitted until 2026-04-16 16:01:21.632940078 +0000 UTC m=+3.029024491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-29nkz" (UniqueName: "kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz") pod "network-check-target-8849t" (UID: "342d6e09-3c3b-470d-b383-b470e1c3a086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:21.134415 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.134391    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2j87\" (UniqueName: \"kubernetes.io/projected/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-kube-api-access-r2j87\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.135024 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.134999    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4sbk\" (UniqueName: \"kubernetes.io/projected/2d945238-4c81-4dc5-bf64-145e503a3eb3-kube-api-access-t4sbk\") pod \"node-resolver-7c442\" (UID: \"2d945238-4c81-4dc5-bf64-145e503a3eb3\") " pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.135514 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135482    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-socket-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.135709 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135678    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/547d3bd9-3b02-47a3-a133-480d22876e5f-socket-dir\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.135783 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135735    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ae8e22-0b95-4a1f-9939-147624118297-agent-certs\") pod \"konnectivity-agent-j5rnw\" (UID: \"05ae8e22-0b95-4a1f-9939-147624118297\") " pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:21.135839 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135795    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-systemd\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.135885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135836    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-cni-bin\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.135939 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135899    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-var-lib-cni-bin\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.135998 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135958    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7c9806f-cdc6-417b-95e3-0749a649dcef-etc-systemd\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.136048 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.135999    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-daemon-config\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.136097 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.136058    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-systemd-units\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.136534 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.136508    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/169cfa67-f1fa-4803-84bc-2219a6384156-systemd-units\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.136618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.136570    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-k8s-cni-cncf-io\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.136678 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.136646    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-host-run-k8s-cni-cncf-io\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.137227 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.137189    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kv7j\" (UniqueName: \"kubernetes.io/projected/547d3bd9-3b02-47a3-a133-480d22876e5f-kube-api-access-8kv7j\") pod \"aws-ebs-csi-driver-node-6g6vj\" (UID: \"547d3bd9-3b02-47a3-a133-480d22876e5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.137320 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.137236    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjk6s\" (UniqueName: \"kubernetes.io/projected/c7c9806f-cdc6-417b-95e3-0749a649dcef-kube-api-access-fjk6s\") pod \"tuned-p6drl\" (UID: \"c7c9806f-cdc6-417b-95e3-0749a649dcef\") " pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.137719 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.137699    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ba1e3eb8-7e0a-4f07-add3-8e05de8932bf-multus-daemon-config\") pod \"multus-xkfxp\" (UID: \"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf\") " pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.138386 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.138359    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xdj\" (UniqueName: \"kubernetes.io/projected/5343bde2-108d-43c0-beaf-40395b901cce-kube-api-access-b9xdj\") pod \"multus-additional-cni-plugins-52kws\" (UID: \"5343bde2-108d-43c0-beaf-40395b901cce\") " pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.138386 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.138389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wt2p\" (UniqueName: \"kubernetes.io/projected/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-kube-api-access-5wt2p\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:21.139581 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.139558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ae8e22-0b95-4a1f-9939-147624118297-agent-certs\") pod \"konnectivity-agent-j5rnw\" (UID: \"05ae8e22-0b95-4a1f-9939-147624118297\") " pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:21.141957 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.141934    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndljn\" (UniqueName: \"kubernetes.io/projected/169cfa67-f1fa-4803-84bc-2219a6384156-kube-api-access-ndljn\") pod \"ovnkube-node-9n7wn\" (UID: \"169cfa67-f1fa-4803-84bc-2219a6384156\") " pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.142285 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.142265    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2st\" (UniqueName: \"kubernetes.io/projected/26fd2187-a513-457a-9763-91f0fdff4ad6-kube-api-access-kh2st\") pod \"node-ca-nwd2d\" (UID: \"26fd2187-a513-457a-9763-91f0fdff4ad6\") " pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.314279 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.314196    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-52kws"
Apr 16 16:01:21.331102 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.331077    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xkfxp"
Apr 16 16:01:21.337758 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.337741    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:21.343832 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.343813    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7c442"
Apr 16 16:01:21.351401 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.351385    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nwd2d"
Apr 16 16:01:21.358964 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.358949    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxkkz"
Apr 16 16:01:21.366596 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.366576    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:21.374135 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.374115    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj"
Apr 16 16:01:21.380654 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.380637    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p6drl"
Apr 16 16:01:21.639402 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.639326    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:21.639402 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:21.639376    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:21.639636 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.639492    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:21.639636 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.639523    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:01:21.639636 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.639538    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:01:21.639636 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.639550    2578 projected.go:194] Error preparing data for projected volume kube-api-access-29nkz for pod openshift-network-diagnostics/network-check-target-8849t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:21.639636 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.639558    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs podName:53da8c88-a72c-4cf8-abde-f3a449cbdf3f nodeName:}" failed. No retries permitted until 2026-04-16 16:01:22.639539122 +0000 UTC m=+4.035623523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs") pod "network-metrics-daemon-pbf54" (UID: "53da8c88-a72c-4cf8-abde-f3a449cbdf3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:21.639861 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:21.639645    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz podName:342d6e09-3c3b-470d-b383-b470e1c3a086 nodeName:}" failed. No retries permitted until 2026-04-16 16:01:22.639607794 +0000 UTC m=+4.035692194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-29nkz" (UniqueName: "kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz") pod "network-check-target-8849t" (UID: "342d6e09-3c3b-470d-b383-b470e1c3a086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:21.765549 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.765523    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1e3eb8_7e0a_4f07_add3_8e05de8932bf.slice/crio-8b931d001c7930795a2caa478cf1d01c76e7b73d4a5483f5c258a5c0c394d982 WatchSource:0}: Error finding container 8b931d001c7930795a2caa478cf1d01c76e7b73d4a5483f5c258a5c0c394d982: Status 404 returned error can't find the container with id 8b931d001c7930795a2caa478cf1d01c76e7b73d4a5483f5c258a5c0c394d982
Apr 16 16:01:21.766121 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.766060    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169cfa67_f1fa_4803_84bc_2219a6384156.slice/crio-5a5c027fbb7ccbfd8c911cb4dda19b68b595b938b4a7f47bd74819086cbea078 WatchSource:0}: Error finding container 5a5c027fbb7ccbfd8c911cb4dda19b68b595b938b4a7f47bd74819086cbea078: Status 404 returned error can't find the container with id 5a5c027fbb7ccbfd8c911cb4dda19b68b595b938b4a7f47bd74819086cbea078
Apr 16 16:01:21.767916 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.767890    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547d3bd9_3b02_47a3_a133_480d22876e5f.slice/crio-af7a27cf48948e1f9a3451821f8f7c0dfe0b30a295c8e98628402f02f2bba586 WatchSource:0}: Error finding container af7a27cf48948e1f9a3451821f8f7c0dfe0b30a295c8e98628402f02f2bba586: Status 404 returned error can't find the container with id af7a27cf48948e1f9a3451821f8f7c0dfe0b30a295c8e98628402f02f2bba586
Apr 16 16:01:21.773535 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.773515    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ae8e22_0b95_4a1f_9939_147624118297.slice/crio-45a6fc011ef21d2e6d7ae88de331c2963014c20fd2f557a6f77ff9186bf68ac8 WatchSource:0}: Error finding container 45a6fc011ef21d2e6d7ae88de331c2963014c20fd2f557a6f77ff9186bf68ac8: Status 404 returned error can't find the container with id 45a6fc011ef21d2e6d7ae88de331c2963014c20fd2f557a6f77ff9186bf68ac8
Apr 16 16:01:21.774355 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.774335    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343a0f51_524e_4bf4_82df_e0253871139f.slice/crio-a75ef9ed2ed539e229b3b092b9ceea8b4ba607fab4e679bc21aaf36337913c64 WatchSource:0}: Error finding container a75ef9ed2ed539e229b3b092b9ceea8b4ba607fab4e679bc21aaf36337913c64: Status 404 returned error can't find the container with id a75ef9ed2ed539e229b3b092b9ceea8b4ba607fab4e679bc21aaf36337913c64
Apr 16 16:01:21.776383 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.776361    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d945238_4c81_4dc5_bf64_145e503a3eb3.slice/crio-c27afbf3f58bf77c463d688ff29b36dda569156ae26ed5ef50e61a46f4130ef3 WatchSource:0}: Error finding container c27afbf3f58bf77c463d688ff29b36dda569156ae26ed5ef50e61a46f4130ef3: Status 404 returned error can't find the container with id c27afbf3f58bf77c463d688ff29b36dda569156ae26ed5ef50e61a46f4130ef3
Apr 16 16:01:21.778084 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.777354    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5343bde2_108d_43c0_beaf_40395b901cce.slice/crio-a58e21a7e0270150bdb1fc93f413ea79b2785d0aefbaca5d6ea164523e98cf07 WatchSource:0}: Error finding container a58e21a7e0270150bdb1fc93f413ea79b2785d0aefbaca5d6ea164523e98cf07: Status 404 returned error can't find the container with id a58e21a7e0270150bdb1fc93f413ea79b2785d0aefbaca5d6ea164523e98cf07
Apr 16 16:01:21.778872 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.778815    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fd2187_a513_457a_9763_91f0fdff4ad6.slice/crio-20a50159dca442237469ddfc6c4a38b9cadb8836b61f7222bc6490dfa81fc27f WatchSource:0}: Error finding container 20a50159dca442237469ddfc6c4a38b9cadb8836b61f7222bc6490dfa81fc27f: Status 404 returned error can't find the container with id 20a50159dca442237469ddfc6c4a38b9cadb8836b61f7222bc6490dfa81fc27f
Apr 16 16:01:21.779635 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:01:21.779616    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c9806f_cdc6_417b_95e3_0749a649dcef.slice/crio-3e4117899282560143d8c24dd12934c28333dc3ef776873ad32db2640476a7cc WatchSource:0}: Error finding container 3e4117899282560143d8c24dd12934c28333dc3ef776873ad32db2640476a7cc: Status 404 returned error can't find the container with id 3e4117899282560143d8c24dd12934c28333dc3ef776873ad32db2640476a7cc
Apr 16 16:01:22.045030 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.044749    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 15:56:20 +0000 UTC" deadline="2028-01-19 15:01:42.072540106 +0000 UTC"
Apr 16 16:01:22.045030 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.044948    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15431h0m20.02759748s"
Apr 16 16:01:22.112006 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.111968    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" event={"ID":"ac3c29e3e9ef322bb0cc9eb018b42ffc","Type":"ContainerStarted","Data":"d4e2841d61a7226116d38aea55f54877e814a65543fa8502584e43304922cebb"}
Apr 16 16:01:22.113134 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.113099    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p6drl" event={"ID":"c7c9806f-cdc6-417b-95e3-0749a649dcef","Type":"ContainerStarted","Data":"3e4117899282560143d8c24dd12934c28333dc3ef776873ad32db2640476a7cc"}
Apr 16 16:01:22.114218 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.114186    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwd2d" event={"ID":"26fd2187-a513-457a-9763-91f0fdff4ad6","Type":"ContainerStarted","Data":"20a50159dca442237469ddfc6c4a38b9cadb8836b61f7222bc6490dfa81fc27f"}
Apr 16 16:01:22.115300 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.115276    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxkkz" event={"ID":"343a0f51-524e-4bf4-82df-e0253871139f","Type":"ContainerStarted","Data":"a75ef9ed2ed539e229b3b092b9ceea8b4ba607fab4e679bc21aaf36337913c64"}
Apr 16 16:01:22.116575 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.116553    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j5rnw" event={"ID":"05ae8e22-0b95-4a1f-9939-147624118297","Type":"ContainerStarted","Data":"45a6fc011ef21d2e6d7ae88de331c2963014c20fd2f557a6f77ff9186bf68ac8"}
Apr 16 16:01:22.118113 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.117892    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"5a5c027fbb7ccbfd8c911cb4dda19b68b595b938b4a7f47bd74819086cbea078"}
Apr 16 16:01:22.119009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.118987    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerStarted","Data":"a58e21a7e0270150bdb1fc93f413ea79b2785d0aefbaca5d6ea164523e98cf07"}
Apr 16 16:01:22.120170 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.120119    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7c442" event={"ID":"2d945238-4c81-4dc5-bf64-145e503a3eb3","Type":"ContainerStarted","Data":"c27afbf3f58bf77c463d688ff29b36dda569156ae26ed5ef50e61a46f4130ef3"}
Apr 16 16:01:22.121177 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.121151    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj" event={"ID":"547d3bd9-3b02-47a3-a133-480d22876e5f","Type":"ContainerStarted","Data":"af7a27cf48948e1f9a3451821f8f7c0dfe0b30a295c8e98628402f02f2bba586"}
Apr 16 16:01:22.122232 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.122209    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xkfxp" event={"ID":"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf","Type":"ContainerStarted","Data":"8b931d001c7930795a2caa478cf1d01c76e7b73d4a5483f5c258a5c0c394d982"}
Apr 16 16:01:22.126954 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.126543    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-41.ec2.internal" podStartSLOduration=2.126527508 podStartE2EDuration="2.126527508s" podCreationTimestamp="2026-04-16 16:01:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:01:22.126116874 +0000 UTC m=+3.522201298" watchObservedRunningTime="2026-04-16 16:01:22.126527508 +0000 UTC m=+3.522611928"
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.648563    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:22.648640    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:22.648778    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:22.648795    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:22.648806    2578 projected.go:194] Error preparing data for projected volume kube-api-access-29nkz for pod openshift-network-diagnostics/network-check-target-8849t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:22.648858    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz podName:342d6e09-3c3b-470d-b383-b470e1c3a086 nodeName:}" failed. No retries permitted until 2026-04-16 16:01:24.648841225 +0000 UTC m=+6.044925628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-29nkz" (UniqueName: "kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz") pod "network-check-target-8849t" (UID: "342d6e09-3c3b-470d-b383-b470e1c3a086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:22.649230    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:22.649308 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:22.649277    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs podName:53da8c88-a72c-4cf8-abde-f3a449cbdf3f nodeName:}" failed. No retries permitted until 2026-04-16 16:01:24.649262187 +0000 UTC m=+6.045346587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs") pod "network-metrics-daemon-pbf54" (UID: "53da8c88-a72c-4cf8-abde-f3a449cbdf3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:23.103942 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:23.103401    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:23.103942 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:23.103546    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:23.104690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:23.104556    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:23.113903 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:23.111562    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:23.137470 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:23.136635    2578 generic.go:358] "Generic (PLEG): container finished" podID="e2d68b261ea36bffca5aca5d127db206" containerID="a011e7fd9dba180b786cc96eca5d846236dfac38fd033a06e23081f5e6349490" exitCode=0
Apr 16 16:01:23.137680 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:23.137617    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" event={"ID":"e2d68b261ea36bffca5aca5d127db206","Type":"ContainerDied","Data":"a011e7fd9dba180b786cc96eca5d846236dfac38fd033a06e23081f5e6349490"}
Apr 16 16:01:24.151292 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:24.151256    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" event={"ID":"e2d68b261ea36bffca5aca5d127db206","Type":"ContainerStarted","Data":"3252de01b3b029e733c77f46ade6c60de11c505e04744c20e31d3b0568f5c47f"}
Apr 16 16:01:24.167737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:24.167690    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-41.ec2.internal" podStartSLOduration=4.167672286 podStartE2EDuration="4.167672286s" podCreationTimestamp="2026-04-16 16:01:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:01:24.166935076 +0000 UTC m=+5.563019497" watchObservedRunningTime="2026-04-16 16:01:24.167672286 +0000 UTC m=+5.563756709"
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:24.666226    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:24.666309    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:24.666433    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:24.666508    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs podName:53da8c88-a72c-4cf8-abde-f3a449cbdf3f nodeName:}" failed. No retries permitted until 2026-04-16 16:01:28.666489307 +0000 UTC m=+10.062573710 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs") pod "network-metrics-daemon-pbf54" (UID: "53da8c88-a72c-4cf8-abde-f3a449cbdf3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:24.666916    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:24.666936    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:24.666948    2578 projected.go:194] Error preparing data for projected volume kube-api-access-29nkz for pod openshift-network-diagnostics/network-check-target-8849t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:24.667047 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:24.666998    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz podName:342d6e09-3c3b-470d-b383-b470e1c3a086 nodeName:}" failed. No retries permitted until 2026-04-16 16:01:28.666980676 +0000 UTC m=+10.063065097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-29nkz" (UniqueName: "kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz") pod "network-check-target-8849t" (UID: "342d6e09-3c3b-470d-b383-b470e1c3a086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:25.103372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:25.102710    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:25.103372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:25.102729    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:25.103372 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:25.102848    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:25.103372 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:25.103276    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:27.104381 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:27.104343    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:27.104841 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:27.104436    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:27.104841 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:27.104731    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:27.104841 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:27.104796    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:28.696994 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:28.696950    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:28.697428 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:28.697039    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:28.697428 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:28.697204    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:01:28.697428 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:28.697224    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:01:28.697428 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:28.697237    2578 projected.go:194] Error preparing data for projected volume kube-api-access-29nkz for pod openshift-network-diagnostics/network-check-target-8849t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:28.697428 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:28.697291    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz podName:342d6e09-3c3b-470d-b383-b470e1c3a086 nodeName:}" failed. No retries permitted until 2026-04-16 16:01:36.697275483 +0000 UTC m=+18.093359899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-29nkz" (UniqueName: "kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz") pod "network-check-target-8849t" (UID: "342d6e09-3c3b-470d-b383-b470e1c3a086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:28.697428 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:28.697356    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:28.697428 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:28.697390    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs podName:53da8c88-a72c-4cf8-abde-f3a449cbdf3f nodeName:}" failed. No retries permitted until 2026-04-16 16:01:36.697379137 +0000 UTC m=+18.093463553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs") pod "network-metrics-daemon-pbf54" (UID: "53da8c88-a72c-4cf8-abde-f3a449cbdf3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:29.103795 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:29.103711    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:29.103795 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:29.103752    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:29.104012 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:29.103850    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:29.104012 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:29.103873    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:31.102838 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:31.102799    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:31.103268 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:31.102936    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:31.103268 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:31.103007    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:31.103268 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:31.103147    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:33.103352 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:33.103317    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:33.103807 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:33.103323    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:33.103807 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:33.103437    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:33.103807 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:33.103563    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:35.102764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:35.102735    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:35.102764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:35.102756    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:35.103231 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:35.102849    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:35.103231 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:35.102939    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:36.753496 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:36.753463    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:36.753890 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:36.753517    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:36.753890 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:36.753552    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:36.753890 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:36.753635    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs podName:53da8c88-a72c-4cf8-abde-f3a449cbdf3f nodeName:}" failed. No retries permitted until 2026-04-16 16:01:52.753612389 +0000 UTC m=+34.149696791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs") pod "network-metrics-daemon-pbf54" (UID: "53da8c88-a72c-4cf8-abde-f3a449cbdf3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:36.753890 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:36.753634    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:01:36.753890 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:36.753658    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:01:36.753890 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:36.753670    2578 projected.go:194] Error preparing data for projected volume kube-api-access-29nkz for pod openshift-network-diagnostics/network-check-target-8849t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:36.753890 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:36.753716    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz podName:342d6e09-3c3b-470d-b383-b470e1c3a086 nodeName:}" failed. No retries permitted until 2026-04-16 16:01:52.753702534 +0000 UTC m=+34.149786932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-29nkz" (UniqueName: "kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz") pod "network-check-target-8849t" (UID: "342d6e09-3c3b-470d-b383-b470e1c3a086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:37.103227 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:37.102733    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:37.103227 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:37.102781    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:37.103227 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:37.102853    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:37.103227 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:37.103012    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:39.103496 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:39.103463    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:39.103887 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:39.103556    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:39.103887 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:39.103620    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:39.103887 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:39.103699    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:40.183168 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.182660    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j5rnw" event={"ID":"05ae8e22-0b95-4a1f-9939-147624118297","Type":"ContainerStarted","Data":"53f6f12e7c25d21fe1d7411a3f2543aa17ec38fe638dc275ce314af907bdae28"}
Apr 16 16:01:40.186691 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.186670    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:01:40.187072 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.187046    2578 generic.go:358] "Generic (PLEG): container finished" podID="169cfa67-f1fa-4803-84bc-2219a6384156" containerID="ec7e93ea16a2b02899f4d3652c5207a6be2bb0d994af8eade7f88aebd866b6c5" exitCode=1
Apr 16 16:01:40.187178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.187112    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"9c9e20744a1815eb1da77cfa91118dd4b673eb5a55a78a98658cc810be9e03c3"}
Apr 16 16:01:40.187178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.187142    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"921a050d1ed7777230f27a7fb87b497d53624f6080b86e20fc295f249c436901"}
Apr 16 16:01:40.187178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.187159    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"ba21f34143f11fd5fa850f225f1ff1fef7abcfd5f5cbd463329e24dfd7ce3b79"}
Apr 16 16:01:40.187178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.187172    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"044d3c10691951efe8c6e7ccb6827fc39b698e7d21a8c0aaa0f66a2036c83c34"}
Apr 16 16:01:40.187348 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.187186    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerDied","Data":"ec7e93ea16a2b02899f4d3652c5207a6be2bb0d994af8eade7f88aebd866b6c5"}
Apr 16 16:01:40.187348 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.187201    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"f854214cab4551453d0e6812aad6c6f0519fce89f2856355648ceed7543e46a8"}
Apr 16 16:01:40.188811 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.188789    2578 generic.go:358] "Generic (PLEG): container finished" podID="5343bde2-108d-43c0-beaf-40395b901cce" containerID="e8379a993bcc863a9c5e508a30a386a4043a8331849226153fa9a3bed0b6cb36" exitCode=0
Apr 16 16:01:40.188881 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.188856    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerDied","Data":"e8379a993bcc863a9c5e508a30a386a4043a8331849226153fa9a3bed0b6cb36"}
Apr 16 16:01:40.191818 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.191796    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7c442" event={"ID":"2d945238-4c81-4dc5-bf64-145e503a3eb3","Type":"ContainerStarted","Data":"0cda0f93468bfc2f45a9cc3b43c6a99e866f480d955bd0a692c123a89a0570c3"}
Apr 16 16:01:40.193296 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.193271    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj" event={"ID":"547d3bd9-3b02-47a3-a133-480d22876e5f","Type":"ContainerStarted","Data":"55c62e96e58a28ac06053d203c24cdbd088a96b903268b3d2064ff3b2187183d"}
Apr 16 16:01:40.194514 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.194492    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xkfxp" event={"ID":"ba1e3eb8-7e0a-4f07-add3-8e05de8932bf","Type":"ContainerStarted","Data":"4c15ad0da654e081c10648495af999c6ee791fb7923c09f32ff29519a87984b8"}
Apr 16 16:01:40.196041 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.195928    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p6drl" event={"ID":"c7c9806f-cdc6-417b-95e3-0749a649dcef","Type":"ContainerStarted","Data":"c484ac890f0ab572d6656640fbb90a70dbd9efd9197a0ee593d6f0c5ec93f84b"}
Apr 16 16:01:40.197349 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.196582    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j5rnw" podStartSLOduration=3.760962527 podStartE2EDuration="21.196570232s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.777657505 +0000 UTC m=+3.173741904" lastFinishedPulling="2026-04-16 16:01:39.213265204 +0000 UTC m=+20.609349609" observedRunningTime="2026-04-16 16:01:40.196376779 +0000 UTC m=+21.592461200" watchObservedRunningTime="2026-04-16 16:01:40.196570232 +0000 UTC m=+21.592654656"
Apr 16 16:01:40.197777 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.197753    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nwd2d" event={"ID":"26fd2187-a513-457a-9763-91f0fdff4ad6","Type":"ContainerStarted","Data":"f685cd3a18eaed8da47aac60e2b8de56f995fa49bdd077145af47cd7de505f8b"}
Apr 16 16:01:40.210574 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.210530    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-7c442" podStartSLOduration=3.77610256 podStartE2EDuration="21.210518988s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.778685175 +0000 UTC m=+3.174769574" lastFinishedPulling="2026-04-16 16:01:39.213101603 +0000 UTC m=+20.609186002" observedRunningTime="2026-04-16 16:01:40.21002818 +0000 UTC m=+21.606112600" watchObservedRunningTime="2026-04-16 16:01:40.210518988 +0000 UTC m=+21.606603408"
Apr 16 16:01:40.229887 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.229850    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p6drl" podStartSLOduration=3.798450216 podStartE2EDuration="21.229842606s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.781577964 +0000 UTC m=+3.177662362" lastFinishedPulling="2026-04-16 16:01:39.212970337 +0000 UTC m=+20.609054752" observedRunningTime="2026-04-16 16:01:40.229566987 +0000 UTC m=+21.625651407" watchObservedRunningTime="2026-04-16 16:01:40.229842606 +0000 UTC m=+21.625927025"
Apr 16 16:01:40.267602 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.267561    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-xkfxp" podStartSLOduration=3.786411631 podStartE2EDuration="21.26755163s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.767820918 +0000 UTC m=+3.163905319" lastFinishedPulling="2026-04-16 16:01:39.248960905 +0000 UTC m=+20.645045318" observedRunningTime="2026-04-16 16:01:40.26723808 +0000 UTC m=+21.663322500" watchObservedRunningTime="2026-04-16 16:01:40.26755163 +0000 UTC m=+21.663636049"
Apr 16 16:01:40.281161 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.281112    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nwd2d" podStartSLOduration=8.691903495 podStartE2EDuration="21.281095173s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.780234539 +0000 UTC m=+3.176318942" lastFinishedPulling="2026-04-16 16:01:34.36942621 +0000 UTC m=+15.765510620" observedRunningTime="2026-04-16 16:01:40.280895964 +0000 UTC m=+21.676980385" watchObservedRunningTime="2026-04-16 16:01:40.281095173 +0000 UTC m=+21.677179593"
Apr 16 16:01:40.815715 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:40.815694    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Apr 16 16:01:41.083704 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.083551    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-16T16:01:40.815710367Z","UUID":"1c94082b-f54c-4c24-8dfd-eecb583fdf42","Handler":null,"Name":"","Endpoint":""}
Apr 16 16:01:41.086120 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.086089    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Apr 16 16:01:41.086285 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.086131    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Apr 16 16:01:41.102929 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.102911    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:41.103026 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.102918    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:41.103094 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:41.103021    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:41.103143 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:41.103108    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:41.201604 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.201563    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj" event={"ID":"547d3bd9-3b02-47a3-a133-480d22876e5f","Type":"ContainerStarted","Data":"b014893b5db104f89e5e7bd6198f399ccddda8f29490323571235157e8154e3d"}
Apr 16 16:01:41.202970 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.202936    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxkkz" event={"ID":"343a0f51-524e-4bf4-82df-e0253871139f","Type":"ContainerStarted","Data":"02aa90e900d1c7cb28cf178c37cc32110202fdb068f6f717a8e889a3c1effd9d"}
Apr 16 16:01:41.217748 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.217706    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rxkkz" podStartSLOduration=4.78157433 podStartE2EDuration="22.217693365s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.776867519 +0000 UTC m=+3.172951916" lastFinishedPulling="2026-04-16 16:01:39.212986552 +0000 UTC m=+20.609070951" observedRunningTime="2026-04-16 16:01:41.217389021 +0000 UTC m=+22.613473443" watchObservedRunningTime="2026-04-16 16:01:41.217693365 +0000 UTC m=+22.613777785"
Apr 16 16:01:41.473047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:41.473001    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:42.207541 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:42.207509    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj" event={"ID":"547d3bd9-3b02-47a3-a133-480d22876e5f","Type":"ContainerStarted","Data":"4f3bfd6769628262ad39cce36d020744fad5fe8d036edae7ca755477dbe9d8a9"}
Apr 16 16:01:43.089631 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.089380    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:43.090146 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.090122    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:43.102984 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.102960    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:43.103104 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:43.103081    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:43.103162 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.103143    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:43.103284 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:43.103264    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:43.212737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.212715    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:01:43.213115 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.213092    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"8a4880168e17c8f7ef920ea84f6dedb2b97e3a5cfc63fe767e5ab0046a67a082"}
Apr 16 16:01:43.213809 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.213791    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j5rnw"
Apr 16 16:01:43.232548 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:43.232500    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g6vj" podStartSLOduration=3.978611564 podStartE2EDuration="24.232485521s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.771931203 +0000 UTC m=+3.168015600" lastFinishedPulling="2026-04-16 16:01:42.025805154 +0000 UTC m=+23.421889557" observedRunningTime="2026-04-16 16:01:43.232001464 +0000 UTC m=+24.628085897" watchObservedRunningTime="2026-04-16 16:01:43.232485521 +0000 UTC m=+24.628569944"
Apr 16 16:01:45.102828 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.102659    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:45.103550 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.102659    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:45.103550 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:45.102914    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:45.103550 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:45.102964    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:45.219672 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.219652    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:01:45.219985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.219964    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"82380934910e8ee21ea8927d3c9ed5bbbed3c53a359050918afbe54967be3900"}
Apr 16 16:01:45.220310 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.220286    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:45.220401 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.220318    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:45.220401 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.220328    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:45.220401 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.220357    2578 scope.go:117] "RemoveContainer" containerID="ec7e93ea16a2b02899f4d3652c5207a6be2bb0d994af8eade7f88aebd866b6c5"
Apr 16 16:01:45.222093 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.222072    2578 generic.go:358] "Generic (PLEG): container finished" podID="5343bde2-108d-43c0-beaf-40395b901cce" containerID="aded116b911629e32ae27f11b9721cdc9ce657247c8c0043ee7c5fb8e58256df" exitCode=0
Apr 16 16:01:45.222193 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.222160    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerDied","Data":"aded116b911629e32ae27f11b9721cdc9ce657247c8c0043ee7c5fb8e58256df"}
Apr 16 16:01:45.236019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.236002    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:45.236142 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:45.236059    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:01:46.226513 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:46.226484    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:01:46.226937 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:46.226869    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" event={"ID":"169cfa67-f1fa-4803-84bc-2219a6384156","Type":"ContainerStarted","Data":"fff593623d82f72c84c8f75669aa48b8cabd24eb2d52f589e254ffbf94b8899b"}
Apr 16 16:01:46.230484 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:46.229582    2578 generic.go:358] "Generic (PLEG): container finished" podID="5343bde2-108d-43c0-beaf-40395b901cce" containerID="ad1ee3fe8ad95deb1a348862ca00510147a4e1a247771a81f909223428d77a60" exitCode=0
Apr 16 16:01:46.230484 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:46.229638    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerDied","Data":"ad1ee3fe8ad95deb1a348862ca00510147a4e1a247771a81f909223428d77a60"}
Apr 16 16:01:46.255253 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:46.255213    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn" podStartSLOduration=9.738523329 podStartE2EDuration="27.255201667s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.769089111 +0000 UTC m=+3.165173516" lastFinishedPulling="2026-04-16 16:01:39.285767454 +0000 UTC m=+20.681851854" observedRunningTime="2026-04-16 16:01:46.253646764 +0000 UTC m=+27.649731185" watchObservedRunningTime="2026-04-16 16:01:46.255201667 +0000 UTC m=+27.651286086"
Apr 16 16:01:47.103224 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:47.103164    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:47.103319 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:47.103165    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:47.103319 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:47.103268    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:47.103408 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:47.103378    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:47.233794 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:47.233766    2578 generic.go:358] "Generic (PLEG): container finished" podID="5343bde2-108d-43c0-beaf-40395b901cce" containerID="72940a7ff179219e6fecc3b6cc4b4459fc578d32069d1b8f4c856aac58b5f2b6" exitCode=0
Apr 16 16:01:47.234172 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:47.233837    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerDied","Data":"72940a7ff179219e6fecc3b6cc4b4459fc578d32069d1b8f4c856aac58b5f2b6"}
Apr 16 16:01:49.103755 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:49.103722    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:49.104189 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:49.103849    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:49.104189 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:49.103887    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:49.104189 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:49.103967    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:51.102932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:51.102896    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:51.103437 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:51.103030    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:51.103437 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:51.103088    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:51.103437 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:51.103221    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:52.771221 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:52.771185    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:52.771778 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:52.771231    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:52.771778 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:52.771319    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:52.771778 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:52.771364    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Apr 16 16:01:52.771778 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:52.771390    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Apr 16 16:01:52.771778 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:52.771404    2578 projected.go:194] Error preparing data for projected volume kube-api-access-29nkz for pod openshift-network-diagnostics/network-check-target-8849t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:52.771778 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:52.771371    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs podName:53da8c88-a72c-4cf8-abde-f3a449cbdf3f nodeName:}" failed. No retries permitted until 2026-04-16 16:02:24.771354934 +0000 UTC m=+66.167439332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs") pod "network-metrics-daemon-pbf54" (UID: "53da8c88-a72c-4cf8-abde-f3a449cbdf3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Apr 16 16:01:52.771778 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:52.771480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz podName:342d6e09-3c3b-470d-b383-b470e1c3a086 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:24.771443111 +0000 UTC m=+66.167527509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-29nkz" (UniqueName: "kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz") pod "network-check-target-8849t" (UID: "342d6e09-3c3b-470d-b383-b470e1c3a086") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Apr 16 16:01:53.102880 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:53.102850    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:53.103001 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:53.102974    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:53.103118 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:53.103023    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:53.103175 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:53.103143    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:53.245533 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:53.245506    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerStarted","Data":"3887923abcea330567768408b8800735a865953c4f2e9ffd43004e9dcb566792"}
Apr 16 16:01:54.249484 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:54.249430    2578 generic.go:358] "Generic (PLEG): container finished" podID="5343bde2-108d-43c0-beaf-40395b901cce" containerID="3887923abcea330567768408b8800735a865953c4f2e9ffd43004e9dcb566792" exitCode=0
Apr 16 16:01:54.249903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:54.249483    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerDied","Data":"3887923abcea330567768408b8800735a865953c4f2e9ffd43004e9dcb566792"}
Apr 16 16:01:55.103335 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:55.103299    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:55.103509 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:55.103313    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:55.103509 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:55.103426    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:55.103509 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:55.103500    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:55.253465 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:55.253419    2578 generic.go:358] "Generic (PLEG): container finished" podID="5343bde2-108d-43c0-beaf-40395b901cce" containerID="fdec8ac8eec63c65d4c0bf95c4b043cf39d54272d21735f084773fbc71319a40" exitCode=0
Apr 16 16:01:55.253818 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:55.253476    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerDied","Data":"fdec8ac8eec63c65d4c0bf95c4b043cf39d54272d21735f084773fbc71319a40"}
Apr 16 16:01:56.259264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:56.259226    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52kws" event={"ID":"5343bde2-108d-43c0-beaf-40395b901cce","Type":"ContainerStarted","Data":"14ca64b1e72d6be97556b23afa9e805ac72461a120ed3ba9c28aa2fd08c7ced6"}
Apr 16 16:01:56.282281 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:56.282243    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-52kws" podStartSLOduration=5.990617848 podStartE2EDuration="37.28222873s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:01:21.779338991 +0000 UTC m=+3.175423404" lastFinishedPulling="2026-04-16 16:01:53.070949885 +0000 UTC m=+34.467034286" observedRunningTime="2026-04-16 16:01:56.281920042 +0000 UTC m=+37.678004461" watchObservedRunningTime="2026-04-16 16:01:56.28222873 +0000 UTC m=+37.678313149"
Apr 16 16:01:57.102653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:57.102617    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:57.102841 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:57.102622    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:57.102841 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:57.102747    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:57.102841 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:57.102799    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:01:58.341659 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:58.341398    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8849t"]
Apr 16 16:01:58.342013 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:58.341750    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:01:58.342013 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:58.341859    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:01:58.344254 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:58.344221    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbf54"]
Apr 16 16:01:58.344354 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:01:58.344301    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:01:58.344398 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:01:58.344375    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:02:00.102764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:00.102730    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:02:00.102764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:00.102742    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:02:00.103414 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:00.102813    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:02:00.103414 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:00.102953    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:02:02.102555 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.102523    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:02:02.102993 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.102523    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:02:02.102993 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:02.102619    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbf54" podUID="53da8c88-a72c-4cf8-abde-f3a449cbdf3f"
Apr 16 16:02:02.102993 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:02.102699    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8849t" podUID="342d6e09-3c3b-470d-b383-b470e1c3a086"
Apr 16 16:02:02.413158 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.413083    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-41.ec2.internal" event="NodeReady"
Apr 16 16:02:02.413266 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.413203    2578 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Apr 16 16:02:02.461802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.461771    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2rp8s"]
Apr 16 16:02:02.463842 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.463824    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.465527 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.465502    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vz6nr"]
Apr 16 16:02:02.466515 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.466494    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jr2th\""
Apr 16 16:02:02.466619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.466600    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Apr 16 16:02:02.466693 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.466635    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Apr 16 16:02:02.467321 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.467304    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.469956 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.469933    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Apr 16 16:02:02.470077 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.469969    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5q2sn\""
Apr 16 16:02:02.470077 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.469940    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Apr 16 16:02:02.470493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.470474    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Apr 16 16:02:02.475363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.475333    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2rp8s"]
Apr 16 16:02:02.479181 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.479158    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vz6nr"]
Apr 16 16:02:02.548328 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.548302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32934c-e56b-47ea-97f7-fb45fd464676-config-volume\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.548433 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.548331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f32934c-e56b-47ea-97f7-fb45fd464676-metrics-tls\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.548433 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.548354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9t8\" (UniqueName: \"kubernetes.io/projected/8f32934c-e56b-47ea-97f7-fb45fd464676-kube-api-access-hg9t8\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.548433 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.548422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f32934c-e56b-47ea-97f7-fb45fd464676-tmp-dir\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.548620 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.548498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a752880f-261b-4efa-a2db-25265bb0a5bb-cert\") pod \"ingress-canary-vz6nr\" (UID: \"a752880f-261b-4efa-a2db-25265bb0a5bb\") " pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.548620 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.548523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghkf\" (UniqueName: \"kubernetes.io/projected/a752880f-261b-4efa-a2db-25265bb0a5bb-kube-api-access-nghkf\") pod \"ingress-canary-vz6nr\" (UID: \"a752880f-261b-4efa-a2db-25265bb0a5bb\") " pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.649508 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.649485    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f32934c-e56b-47ea-97f7-fb45fd464676-tmp-dir\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.649575 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.649526    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a752880f-261b-4efa-a2db-25265bb0a5bb-cert\") pod \"ingress-canary-vz6nr\" (UID: \"a752880f-261b-4efa-a2db-25265bb0a5bb\") " pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.649575 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.649544    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nghkf\" (UniqueName: \"kubernetes.io/projected/a752880f-261b-4efa-a2db-25265bb0a5bb-kube-api-access-nghkf\") pod \"ingress-canary-vz6nr\" (UID: \"a752880f-261b-4efa-a2db-25265bb0a5bb\") " pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.649641 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.649591    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32934c-e56b-47ea-97f7-fb45fd464676-config-volume\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.649641 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.649617    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f32934c-e56b-47ea-97f7-fb45fd464676-metrics-tls\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.649733 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.649640    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9t8\" (UniqueName: \"kubernetes.io/projected/8f32934c-e56b-47ea-97f7-fb45fd464676-kube-api-access-hg9t8\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.649829 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.649811    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f32934c-e56b-47ea-97f7-fb45fd464676-tmp-dir\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.650136 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.650111    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f32934c-e56b-47ea-97f7-fb45fd464676-config-volume\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.653824 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.653798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f32934c-e56b-47ea-97f7-fb45fd464676-metrics-tls\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.653892 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.653836    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a752880f-261b-4efa-a2db-25265bb0a5bb-cert\") pod \"ingress-canary-vz6nr\" (UID: \"a752880f-261b-4efa-a2db-25265bb0a5bb\") " pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.658664 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.658618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9t8\" (UniqueName: \"kubernetes.io/projected/8f32934c-e56b-47ea-97f7-fb45fd464676-kube-api-access-hg9t8\") pod \"dns-default-2rp8s\" (UID: \"8f32934c-e56b-47ea-97f7-fb45fd464676\") " pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.659419 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.659396    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghkf\" (UniqueName: \"kubernetes.io/projected/a752880f-261b-4efa-a2db-25265bb0a5bb-kube-api-access-nghkf\") pod \"ingress-canary-vz6nr\" (UID: \"a752880f-261b-4efa-a2db-25265bb0a5bb\") " pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.707830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.707778    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"]
Apr 16 16:02:02.710141 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.710127    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"
Apr 16 16:02:02.712569 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.712546    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lwsrf\""
Apr 16 16:02:02.712698 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.712679    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Apr 16 16:02:02.720677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.720644    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"]
Apr 16 16:02:02.749940 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.749920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/614a4262-9bed-4287-bdb0-5d151f9397bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-cjd4p\" (UID: \"614a4262-9bed-4287-bdb0-5d151f9397bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"
Apr 16 16:02:02.774990 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.774969    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:02.781683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.781667    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vz6nr"
Apr 16 16:02:02.792299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.791792    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"]
Apr 16 16:02:02.794688 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.794622    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"
Apr 16 16:02:02.797606 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.797327    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Apr 16 16:02:02.797606 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.797496    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Apr 16 16:02:02.797789 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.797613    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t78pl\""
Apr 16 16:02:02.807574 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.807548    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"]
Apr 16 16:02:02.851238 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.851129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvzbj\" (UniqueName: \"kubernetes.io/projected/7876d91d-8e38-415b-b557-f51d2d0c907c-kube-api-access-gvzbj\") pod \"migrator-64d4d94569-nl8rd\" (UID: \"7876d91d-8e38-415b-b557-f51d2d0c907c\") " pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"
Apr 16 16:02:02.851720 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.851254    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/614a4262-9bed-4287-bdb0-5d151f9397bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-cjd4p\" (UID: \"614a4262-9bed-4287-bdb0-5d151f9397bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"
Apr 16 16:02:02.856477 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.856090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/614a4262-9bed-4287-bdb0-5d151f9397bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-cjd4p\" (UID: \"614a4262-9bed-4287-bdb0-5d151f9397bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"
Apr 16 16:02:02.916028 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.916004    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vz6nr"]
Apr 16 16:02:02.919081 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:02.919055    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda752880f_261b_4efa_a2db_25265bb0a5bb.slice/crio-3c199d8ec9282b2aea034eb635ace0e7c8c06ddd08ae91bdf52231b635762248 WatchSource:0}: Error finding container 3c199d8ec9282b2aea034eb635ace0e7c8c06ddd08ae91bdf52231b635762248: Status 404 returned error can't find the container with id 3c199d8ec9282b2aea034eb635ace0e7c8c06ddd08ae91bdf52231b635762248
Apr 16 16:02:02.935481 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.935438    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2rp8s"]
Apr 16 16:02:02.938833 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:02.938811    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f32934c_e56b_47ea_97f7_fb45fd464676.slice/crio-07ba5a29904fba74f4e219ae6833ae9e0c77e699a3ace6c350996b74ad288293 WatchSource:0}: Error finding container 07ba5a29904fba74f4e219ae6833ae9e0c77e699a3ace6c350996b74ad288293: Status 404 returned error can't find the container with id 07ba5a29904fba74f4e219ae6833ae9e0c77e699a3ace6c350996b74ad288293
Apr 16 16:02:02.951617 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.951599    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gvzbj\" (UniqueName: \"kubernetes.io/projected/7876d91d-8e38-415b-b557-f51d2d0c907c-kube-api-access-gvzbj\") pod \"migrator-64d4d94569-nl8rd\" (UID: \"7876d91d-8e38-415b-b557-f51d2d0c907c\") " pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"
Apr 16 16:02:02.971882 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:02.971857    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvzbj\" (UniqueName: \"kubernetes.io/projected/7876d91d-8e38-415b-b557-f51d2d0c907c-kube-api-access-gvzbj\") pod \"migrator-64d4d94569-nl8rd\" (UID: \"7876d91d-8e38-415b-b557-f51d2d0c907c\") " pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"
Apr 16 16:02:03.018605 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.018581    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"
Apr 16 16:02:03.106595 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.106574    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"
Apr 16 16:02:03.134116 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.134083    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"]
Apr 16 16:02:03.137811 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:03.137786    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614a4262_9bed_4287_bdb0_5d151f9397bc.slice/crio-fcc3207e63eca4496538a031fcf4fc1d2865e224bfb07fc5f0808418bb72d8e2 WatchSource:0}: Error finding container fcc3207e63eca4496538a031fcf4fc1d2865e224bfb07fc5f0808418bb72d8e2: Status 404 returned error can't find the container with id fcc3207e63eca4496538a031fcf4fc1d2865e224bfb07fc5f0808418bb72d8e2
Apr 16 16:02:03.239313 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.239254    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd"]
Apr 16 16:02:03.241786 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:03.241762    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7876d91d_8e38_415b_b557_f51d2d0c907c.slice/crio-7f17638236961886465ab5236361514e8dc51f350023f8e0f9910f697f357879 WatchSource:0}: Error finding container 7f17638236961886465ab5236361514e8dc51f350023f8e0f9910f697f357879: Status 404 returned error can't find the container with id 7f17638236961886465ab5236361514e8dc51f350023f8e0f9910f697f357879
Apr 16 16:02:03.271985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.271955    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p" event={"ID":"614a4262-9bed-4287-bdb0-5d151f9397bc","Type":"ContainerStarted","Data":"fcc3207e63eca4496538a031fcf4fc1d2865e224bfb07fc5f0808418bb72d8e2"}
Apr 16 16:02:03.272852 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.272830    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd" event={"ID":"7876d91d-8e38-415b-b557-f51d2d0c907c","Type":"ContainerStarted","Data":"7f17638236961886465ab5236361514e8dc51f350023f8e0f9910f697f357879"}
Apr 16 16:02:03.273724 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.273702    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vz6nr" event={"ID":"a752880f-261b-4efa-a2db-25265bb0a5bb","Type":"ContainerStarted","Data":"3c199d8ec9282b2aea034eb635ace0e7c8c06ddd08ae91bdf52231b635762248"}
Apr 16 16:02:03.274610 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:03.274584    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rp8s" event={"ID":"8f32934c-e56b-47ea-97f7-fb45fd464676","Type":"ContainerStarted","Data":"07ba5a29904fba74f4e219ae6833ae9e0c77e699a3ace6c350996b74ad288293"}
Apr 16 16:02:04.103529 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:04.103492    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:02:04.103762 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:04.103495    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:02:04.109253 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:04.108717    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 16 16:02:04.109253 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:04.108939    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 16 16:02:04.109253 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:04.109063    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4vs75\""
Apr 16 16:02:04.109253 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:04.109139    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x4n9w\""
Apr 16 16:02:04.109885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:04.109357    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 16 16:02:05.816654 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:05.816631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7c442_2d945238-4c81-4dc5-bf64-145e503a3eb3/dns-node-resolver/0.log"
Apr 16 16:02:07.219391 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.219223    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwd2d_26fd2187-a513-457a-9763-91f0fdff4ad6/node-ca/0.log"
Apr 16 16:02:07.284607 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.284575    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p" event={"ID":"614a4262-9bed-4287-bdb0-5d151f9397bc","Type":"ContainerStarted","Data":"b46b078e265bf21bfaaba4e2edf5e69440d6d807135fa234a5548d8480969d15"}
Apr 16 16:02:07.284773 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.284755    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"
Apr 16 16:02:07.286292 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.286242    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd" event={"ID":"7876d91d-8e38-415b-b557-f51d2d0c907c","Type":"ContainerStarted","Data":"2671b14dad8d0705c40293f07274492df3cc75a28d3781517f75cbec3806fb05"}
Apr 16 16:02:07.286292 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.286278    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd" event={"ID":"7876d91d-8e38-415b-b557-f51d2d0c907c","Type":"ContainerStarted","Data":"fd27bed09a68afad67d5b84c1e85f18d71523450eb74fcdaf6a7dcc8783c2554"}
Apr 16 16:02:07.287522 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.287502    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vz6nr" event={"ID":"a752880f-261b-4efa-a2db-25265bb0a5bb","Type":"ContainerStarted","Data":"d15cf151c0fa3ed5ee3c8beaa0b15ef56d7fe75d4a0d69ce1a8897fa62969a8f"}
Apr 16 16:02:07.289175 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.289152    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rp8s" event={"ID":"8f32934c-e56b-47ea-97f7-fb45fd464676","Type":"ContainerStarted","Data":"6d7b2b58a4e777fba6f3c82fe7db6252c2d1ec69364493bcf362ed0dc5f4d0f0"}
Apr 16 16:02:07.289371 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.289182    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rp8s" event={"ID":"8f32934c-e56b-47ea-97f7-fb45fd464676","Type":"ContainerStarted","Data":"81e7b689488264eac12e093acdb4f8ead65f0dd72d49d0c98652e1128070b568"}
Apr 16 16:02:07.289371 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.289300    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:07.290128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.290102    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p"
Apr 16 16:02:07.321156 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.321117    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-2rp8s" podStartSLOduration=1.919564718 podStartE2EDuration="5.321106261s" podCreationTimestamp="2026-04-16 16:02:02 +0000 UTC" firstStartedPulling="2026-04-16 16:02:02.940428558 +0000 UTC m=+44.336512955" lastFinishedPulling="2026-04-16 16:02:06.341970097 +0000 UTC m=+47.738054498" observedRunningTime="2026-04-16 16:02:07.32041869 +0000 UTC m=+48.716503110" watchObservedRunningTime="2026-04-16 16:02:07.321106261 +0000 UTC m=+48.717190681"
Apr 16 16:02:07.321245 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.321192    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-cjd4p" podStartSLOduration=2.118934233 podStartE2EDuration="5.321188757s" podCreationTimestamp="2026-04-16 16:02:02 +0000 UTC" firstStartedPulling="2026-04-16 16:02:03.139616915 +0000 UTC m=+44.535701315" lastFinishedPulling="2026-04-16 16:02:06.341871442 +0000 UTC m=+47.737955839" observedRunningTime="2026-04-16 16:02:07.304633294 +0000 UTC m=+48.700717716" watchObservedRunningTime="2026-04-16 16:02:07.321188757 +0000 UTC m=+48.717273176"
Apr 16 16:02:07.335576 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.335531    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vz6nr" podStartSLOduration=1.914319565 podStartE2EDuration="5.335519151s" podCreationTimestamp="2026-04-16 16:02:02 +0000 UTC" firstStartedPulling="2026-04-16 16:02:02.920719774 +0000 UTC m=+44.316804172" lastFinishedPulling="2026-04-16 16:02:06.341919356 +0000 UTC m=+47.738003758" observedRunningTime="2026-04-16 16:02:07.335268495 +0000 UTC m=+48.731352915" watchObservedRunningTime="2026-04-16 16:02:07.335519151 +0000 UTC m=+48.731603571"
Apr 16 16:02:07.363691 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.363654    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-64d4d94569-nl8rd" podStartSLOduration=2.262562807 podStartE2EDuration="5.36364338s" podCreationTimestamp="2026-04-16 16:02:02 +0000 UTC" firstStartedPulling="2026-04-16 16:02:03.243659785 +0000 UTC m=+44.639744183" lastFinishedPulling="2026-04-16 16:02:06.344740354 +0000 UTC m=+47.740824756" observedRunningTime="2026-04-16 16:02:07.362868196 +0000 UTC m=+48.758952616" watchObservedRunningTime="2026-04-16 16:02:07.36364338 +0000 UTC m=+48.759727799"
Apr 16 16:02:07.780330 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.780299    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-rz29f"]
Apr 16 16:02:07.818728 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.818698    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-rz29f"]
Apr 16 16:02:07.818849 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.818822    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.821731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.821711    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Apr 16 16:02:07.822764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.822745    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Apr 16 16:02:07.822764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.822760    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Apr 16 16:02:07.822897 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.822767    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Apr 16 16:02:07.822897 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.822745    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Apr 16 16:02:07.822897 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.822745    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-768hf\""
Apr 16 16:02:07.888176 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.888155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.888283 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.888188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-metrics-client-ca\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.888283 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.888205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5s4\" (UniqueName: \"kubernetes.io/projected/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-kube-api-access-sq5s4\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.888351 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.888282    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.989273 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.989247    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.989375 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.989290    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-metrics-client-ca\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.989375 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.989321    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5s4\" (UniqueName: \"kubernetes.io/projected/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-kube-api-access-sq5s4\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.989375 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.989369    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.989897 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.989876    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-metrics-client-ca\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.993026 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.992904    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.993119 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.992924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-prometheus-operator-tls\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:07.998186 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:07.998164    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5s4\" (UniqueName: \"kubernetes.io/projected/ab0a2370-e936-4e4f-94aa-9dc341b4ef4c-kube-api-access-sq5s4\") pod \"prometheus-operator-78f957474d-rz29f\" (UID: \"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c\") " pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:08.019210 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:08.019190    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vz6nr_a752880f-261b-4efa-a2db-25265bb0a5bb/serve-healthcheck-canary/0.log"
Apr 16 16:02:08.128234 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:08.128174    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f"
Apr 16 16:02:08.216744 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:08.216707    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-64d4d94569-nl8rd_7876d91d-8e38-415b-b557-f51d2d0c907c/migrator/0.log"
Apr 16 16:02:08.241887 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:08.241861    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-78f957474d-rz29f"]
Apr 16 16:02:08.245413 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:08.245390    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0a2370_e936_4e4f_94aa_9dc341b4ef4c.slice/crio-3e80a70430decf90dd8e9191dbb87caa3c9672172137cfb2f9fbf37f545a3521 WatchSource:0}: Error finding container 3e80a70430decf90dd8e9191dbb87caa3c9672172137cfb2f9fbf37f545a3521: Status 404 returned error can't find the container with id 3e80a70430decf90dd8e9191dbb87caa3c9672172137cfb2f9fbf37f545a3521
Apr 16 16:02:08.295500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:08.295440    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f" event={"ID":"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c","Type":"ContainerStarted","Data":"3e80a70430decf90dd8e9191dbb87caa3c9672172137cfb2f9fbf37f545a3521"}
Apr 16 16:02:08.419353 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:08.419286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-64d4d94569-nl8rd_7876d91d-8e38-415b-b557-f51d2d0c907c/graceful-termination/0.log"
Apr 16 16:02:10.302499 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:10.302257    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f" event={"ID":"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c","Type":"ContainerStarted","Data":"dc6ea75bcc53d85a93f46bf40efc0a7825eb39cfc98f30b235a84bfbf3bee46e"}
Apr 16 16:02:10.302499 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:10.302466    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f" event={"ID":"ab0a2370-e936-4e4f-94aa-9dc341b4ef4c","Type":"ContainerStarted","Data":"642038d04cf3dd72bfc4bcad884376c58b5c09be47374ad92d8a1716aa00697d"}
Apr 16 16:02:10.319865 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:10.319813    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-78f957474d-rz29f" podStartSLOduration=1.573307 podStartE2EDuration="3.319798795s" podCreationTimestamp="2026-04-16 16:02:07 +0000 UTC" firstStartedPulling="2026-04-16 16:02:08.247367805 +0000 UTC m=+49.643452209" lastFinishedPulling="2026-04-16 16:02:09.9938596 +0000 UTC m=+51.389944004" observedRunningTime="2026-04-16 16:02:10.318186846 +0000 UTC m=+51.714271265" watchObservedRunningTime="2026-04-16 16:02:10.319798795 +0000 UTC m=+51.715883214"
Apr 16 16:02:12.132830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.132795    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"]
Apr 16 16:02:12.141099 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.141083    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.143484 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.143436    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Apr 16 16:02:12.143484 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.143476    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Apr 16 16:02:12.143681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.143574    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qlcjq\""
Apr 16 16:02:12.146389 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.146367    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"]
Apr 16 16:02:12.179377 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.179347    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9v8pw"]
Apr 16 16:02:12.190931 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.190904    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.193419 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.193403    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Apr 16 16:02:12.193681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.193669    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Apr 16 16:02:12.193753 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.193736    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rqdzt\""
Apr 16 16:02:12.193855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.193842    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Apr 16 16:02:12.219210 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db13baef-3ba8-4571-b68f-b71473a7aa37-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.219310 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-accelerators-collector-config\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219310 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db13baef-3ba8-4571-b68f-b71473a7aa37-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.219310 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-root\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219310 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-textfile\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vhnr\" (UniqueName: \"kubernetes.io/projected/2b5bff59-222c-4e8a-a42e-bedbd5295241-kube-api-access-7vhnr\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-sys\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b5bff59-222c-4e8a-a42e-bedbd5295241-metrics-client-ca\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zphqv\" (UniqueName: \"kubernetes.io/projected/db13baef-3ba8-4571-b68f-b71473a7aa37-kube-api-access-zphqv\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.219577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-wtmp\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-tls\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.219577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.219550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db13baef-3ba8-4571-b68f-b71473a7aa37-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.320178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320141    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-tls\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320181    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db13baef-3ba8-4571-b68f-b71473a7aa37-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.320356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320218    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db13baef-3ba8-4571-b68f-b71473a7aa37-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.320356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320248    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-accelerators-collector-config\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320274    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db13baef-3ba8-4571-b68f-b71473a7aa37-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.320356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320301    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-root\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320356 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320348    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-textfile\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320366    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7vhnr\" (UniqueName: \"kubernetes.io/projected/2b5bff59-222c-4e8a-a42e-bedbd5295241-kube-api-access-7vhnr\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:12.320387    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320399    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-sys\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320423    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b5bff59-222c-4e8a-a42e-bedbd5295241-metrics-client-ca\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320471    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zphqv\" (UniqueName: \"kubernetes.io/projected/db13baef-3ba8-4571-b68f-b71473a7aa37-kube-api-access-zphqv\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:12.320488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-tls podName:2b5bff59-222c-4e8a-a42e-bedbd5295241 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:12.820468139 +0000 UTC m=+54.216552562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-tls") pod "node-exporter-9v8pw" (UID: "2b5bff59-222c-4e8a-a42e-bedbd5295241") : secret "node-exporter-tls" not found
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320535    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-wtmp\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.320668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320565    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.321027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320665    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-sys\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.321027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320394    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-root\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.321027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.320842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-wtmp\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.321168 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.321033    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-textfile\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.321168 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.321150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-accelerators-collector-config\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.321247 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.321167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db13baef-3ba8-4571-b68f-b71473a7aa37-metrics-client-ca\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.321358 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.321338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b5bff59-222c-4e8a-a42e-bedbd5295241-metrics-client-ca\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.324324 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.324303    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.324458 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.324353    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db13baef-3ba8-4571-b68f-b71473a7aa37-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.324458 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.324392    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db13baef-3ba8-4571-b68f-b71473a7aa37-openshift-state-metrics-tls\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.329127 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.329105    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vhnr\" (UniqueName: \"kubernetes.io/projected/2b5bff59-222c-4e8a-a42e-bedbd5295241-kube-api-access-7vhnr\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.330157 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.330136    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zphqv\" (UniqueName: \"kubernetes.io/projected/db13baef-3ba8-4571-b68f-b71473a7aa37-kube-api-access-zphqv\") pod \"openshift-state-metrics-5669946b84-w6c2l\" (UID: \"db13baef-3ba8-4571-b68f-b71473a7aa37\") " pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.451143 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.451064    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"
Apr 16 16:02:12.570902 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.570871    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l"]
Apr 16 16:02:12.575662 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:12.575628    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb13baef_3ba8_4571_b68f_b71473a7aa37.slice/crio-703ce5c78da84cdfbdb26f6ef9d483d32844da253426e1581bc8de65956e77a2 WatchSource:0}: Error finding container 703ce5c78da84cdfbdb26f6ef9d483d32844da253426e1581bc8de65956e77a2: Status 404 returned error can't find the container with id 703ce5c78da84cdfbdb26f6ef9d483d32844da253426e1581bc8de65956e77a2
Apr 16 16:02:12.825763 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.825561    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-tls\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:12.828871 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:12.828847    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b5bff59-222c-4e8a-a42e-bedbd5295241-node-exporter-tls\") pod \"node-exporter-9v8pw\" (UID: \"2b5bff59-222c-4e8a-a42e-bedbd5295241\") " pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:13.099277 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.099243    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9v8pw"
Apr 16 16:02:13.108717 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:13.108677    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5bff59_222c_4e8a_a42e_bedbd5295241.slice/crio-91897ee73cffec20576962171c182ddd816c3a8d9b7d35c95de5713caae1b5db WatchSource:0}: Error finding container 91897ee73cffec20576962171c182ddd816c3a8d9b7d35c95de5713caae1b5db: Status 404 returned error can't find the container with id 91897ee73cffec20576962171c182ddd816c3a8d9b7d35c95de5713caae1b5db
Apr 16 16:02:13.186365 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.186336    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:02:13.220493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.220467    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:02:13.220628 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.220593    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.223312 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223280    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Apr 16 16:02:13.223312 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223307    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Apr 16 16:02:13.223507 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223394    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Apr 16 16:02:13.223507 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223408    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Apr 16 16:02:13.223507 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223485    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Apr 16 16:02:13.223638 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223508    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Apr 16 16:02:13.223638 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223632    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Apr 16 16:02:13.223723 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223711    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Apr 16 16:02:13.223768 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223712    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Apr 16 16:02:13.223768 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.223733    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5gsrq\""
Apr 16 16:02:13.309979 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.309932    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9v8pw" event={"ID":"2b5bff59-222c-4e8a-a42e-bedbd5295241","Type":"ContainerStarted","Data":"91897ee73cffec20576962171c182ddd816c3a8d9b7d35c95de5713caae1b5db"}
Apr 16 16:02:13.311714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.311678    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l" event={"ID":"db13baef-3ba8-4571-b68f-b71473a7aa37","Type":"ContainerStarted","Data":"4e2d2adcb372e0fc2a5bcffd6a57b2ed11c74f7af66b36b2d1a1b2badf2830c6"}
Apr 16 16:02:13.311714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.311708    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l" event={"ID":"db13baef-3ba8-4571-b68f-b71473a7aa37","Type":"ContainerStarted","Data":"743a8a9e515e0046d9bd3cf83c13a417a57c6812ff0f4bbd8643cca99e8989cc"}
Apr 16 16:02:13.311714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.311719    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l" event={"ID":"db13baef-3ba8-4571-b68f-b71473a7aa37","Type":"ContainerStarted","Data":"703ce5c78da84cdfbdb26f6ef9d483d32844da253426e1581bc8de65956e77a2"}
Apr 16 16:02:13.327939 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.327915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328023 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.327944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328023 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.327987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-web-config\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328023 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpqg\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-kube-api-access-nqpqg\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328241 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-config-out\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328290 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328290 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328360 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328360 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.328432 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.328375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.430903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.430817    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-config-out\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.430903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.430853    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.430903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.430871    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431012    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431057    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431087    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431126    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431154    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431206    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-web-config\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431234    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431269    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpqg\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-kube-api-access-nqpqg\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431303    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.431334    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.431776 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:13.431208    2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Apr 16 16:02:13.431918 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:02:13.431907    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls podName:a9cd60df-ec8c-45db-8750-898a26ba4196 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:13.931880153 +0000 UTC m=+55.327964554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196") : secret "alertmanager-main-tls" not found
Apr 16 16:02:13.433080 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.432419    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.433080 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.432939    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.433080 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.432969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.434385 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.434362    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.436035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.436010    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.436610 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.436327    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-config-out\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.436610 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.436569    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-web-config\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.436988 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.436944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.437506 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.437484    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.438126 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.438102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.441232 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.441211    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpqg\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-kube-api-access-nqpqg\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.444576 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.444535    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.935936 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.935903    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:13.938370 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:13.938343    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:14.130826 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:14.130786    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:02:14.523157 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:14.523050    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:02:14.525303 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:14.525279    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cd60df_ec8c_45db_8750_898a26ba4196.slice/crio-2cad4d3a524b937827c74bfba19adf6b81ce287414b3bfe545fca28552bf1751 WatchSource:0}: Error finding container 2cad4d3a524b937827c74bfba19adf6b81ce287414b3bfe545fca28552bf1751: Status 404 returned error can't find the container with id 2cad4d3a524b937827c74bfba19adf6b81ce287414b3bfe545fca28552bf1751
Apr 16 16:02:15.319671 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:15.319634    2578 generic.go:358] "Generic (PLEG): container finished" podID="2b5bff59-222c-4e8a-a42e-bedbd5295241" containerID="f122d865de9b0c73e549b9207f7b9724172ac6d0dbd6354ef0cc4c2a444ec7dc" exitCode=0
Apr 16 16:02:15.319862 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:15.319723    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9v8pw" event={"ID":"2b5bff59-222c-4e8a-a42e-bedbd5295241","Type":"ContainerDied","Data":"f122d865de9b0c73e549b9207f7b9724172ac6d0dbd6354ef0cc4c2a444ec7dc"}
Apr 16 16:02:15.322145 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:15.322113    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l" event={"ID":"db13baef-3ba8-4571-b68f-b71473a7aa37","Type":"ContainerStarted","Data":"e42a805c50fd955960501a5489855a42b0b4fe2911ac8568fbccaa60e8b4b37c"}
Apr 16 16:02:15.323492 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:15.323469    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerStarted","Data":"2cad4d3a524b937827c74bfba19adf6b81ce287414b3bfe545fca28552bf1751"}
Apr 16 16:02:15.354889 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:15.354851    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-5669946b84-w6c2l" podStartSLOduration=1.725205267 podStartE2EDuration="3.354835974s" podCreationTimestamp="2026-04-16 16:02:12 +0000 UTC" firstStartedPulling="2026-04-16 16:02:12.769138335 +0000 UTC m=+54.165222737" lastFinishedPulling="2026-04-16 16:02:14.398769043 +0000 UTC m=+55.794853444" observedRunningTime="2026-04-16 16:02:15.353487089 +0000 UTC m=+56.749571510" watchObservedRunningTime="2026-04-16 16:02:15.354835974 +0000 UTC m=+56.750920399"
Apr 16 16:02:16.327592 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.327549    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9v8pw" event={"ID":"2b5bff59-222c-4e8a-a42e-bedbd5295241","Type":"ContainerStarted","Data":"11b7206fb6a236d54fcae93eb4344eb6edf530933bb7595408762aa8dda136c4"}
Apr 16 16:02:16.328066 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.327598    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9v8pw" event={"ID":"2b5bff59-222c-4e8a-a42e-bedbd5295241","Type":"ContainerStarted","Data":"be0d1adf926a4ecd1e9558b10fb35c72e5d2edc0fe004f8c545878024b296cc1"}
Apr 16 16:02:16.328850 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.328830    2578 generic.go:358] "Generic (PLEG): container finished" podID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerID="6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f" exitCode=0
Apr 16 16:02:16.328952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.328908    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f"}
Apr 16 16:02:16.348805 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.348762    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9v8pw" podStartSLOduration=3.058812493 podStartE2EDuration="4.348748262s" podCreationTimestamp="2026-04-16 16:02:12 +0000 UTC" firstStartedPulling="2026-04-16 16:02:13.110256102 +0000 UTC m=+54.506340500" lastFinishedPulling="2026-04-16 16:02:14.400191871 +0000 UTC m=+55.796276269" observedRunningTime="2026-04-16 16:02:16.348186534 +0000 UTC m=+57.744270957" watchObservedRunningTime="2026-04-16 16:02:16.348748262 +0000 UTC m=+57.744832681"
Apr 16 16:02:16.518591 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.518557    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55987d9d7d-8qs6k"]
Apr 16 16:02:16.521787 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.521768    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.524851 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.524831    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Apr 16 16:02:16.524962 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.524846    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zrgt9\""
Apr 16 16:02:16.524962 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.524913    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f8v1m7ef69l4a\""
Apr 16 16:02:16.525165 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.525149    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Apr 16 16:02:16.525232 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.525173    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Apr 16 16:02:16.525492 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.525478    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Apr 16 16:02:16.533714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.533690    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55987d9d7d-8qs6k"]
Apr 16 16:02:16.558260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.558236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95d7b734-0366-4e59-85ac-abbc5e0a18cd-audit-log\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.558373 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.558297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95d7b734-0366-4e59-85ac-abbc5e0a18cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.558373 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.558360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-secret-metrics-server-tls\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.558484 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.558387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-client-ca-bundle\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.558484 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.558411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv79m\" (UniqueName: \"kubernetes.io/projected/95d7b734-0366-4e59-85ac-abbc5e0a18cd-kube-api-access-bv79m\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.558589 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.558505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-secret-metrics-server-client-certs\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.558589 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.558559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95d7b734-0366-4e59-85ac-abbc5e0a18cd-metrics-server-audit-profiles\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.659925 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.659831    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-secret-metrics-server-tls\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.659925 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.659868    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-client-ca-bundle\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.659925 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.659891    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bv79m\" (UniqueName: \"kubernetes.io/projected/95d7b734-0366-4e59-85ac-abbc5e0a18cd-kube-api-access-bv79m\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.660190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.659948    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-secret-metrics-server-client-certs\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.660190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.660004    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95d7b734-0366-4e59-85ac-abbc5e0a18cd-metrics-server-audit-profiles\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.660190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.660055    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95d7b734-0366-4e59-85ac-abbc5e0a18cd-audit-log\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.660190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.660126    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95d7b734-0366-4e59-85ac-abbc5e0a18cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.660506 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.660485    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95d7b734-0366-4e59-85ac-abbc5e0a18cd-audit-log\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.660823 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.660800    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95d7b734-0366-4e59-85ac-abbc5e0a18cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.661407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.661385    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95d7b734-0366-4e59-85ac-abbc5e0a18cd-metrics-server-audit-profiles\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.662428 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.662409    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-secret-metrics-server-client-certs\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.662653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.662631    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-secret-metrics-server-tls\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.662716 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.662650    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95d7b734-0366-4e59-85ac-abbc5e0a18cd-client-ca-bundle\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.668543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.668523    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv79m\" (UniqueName: \"kubernetes.io/projected/95d7b734-0366-4e59-85ac-abbc5e0a18cd-kube-api-access-bv79m\") pod \"metrics-server-55987d9d7d-8qs6k\" (UID: \"95d7b734-0366-4e59-85ac-abbc5e0a18cd\") " pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.830533 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.830498    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:16.966185 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:16.965981    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55987d9d7d-8qs6k"]
Apr 16 16:02:16.969228 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:16.969199    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d7b734_0366_4e59_85ac_abbc5e0a18cd.slice/crio-6aaaeb8ab80cea2abd140e6f0363eb131660c7a773b91a9d4ffeb8d8a81e6a43 WatchSource:0}: Error finding container 6aaaeb8ab80cea2abd140e6f0363eb131660c7a773b91a9d4ffeb8d8a81e6a43: Status 404 returned error can't find the container with id 6aaaeb8ab80cea2abd140e6f0363eb131660c7a773b91a9d4ffeb8d8a81e6a43
Apr 16 16:02:17.246904 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:17.246875    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9n7wn"
Apr 16 16:02:17.297777 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:17.297749    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2rp8s"
Apr 16 16:02:17.335282 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:17.335249    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k" event={"ID":"95d7b734-0366-4e59-85ac-abbc5e0a18cd","Type":"ContainerStarted","Data":"6aaaeb8ab80cea2abd140e6f0363eb131660c7a773b91a9d4ffeb8d8a81e6a43"}
Apr 16 16:02:18.344409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:18.344321    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerStarted","Data":"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"}
Apr 16 16:02:18.344409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:18.344367    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerStarted","Data":"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"}
Apr 16 16:02:18.344409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:18.344384    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerStarted","Data":"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"}
Apr 16 16:02:18.347301 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:18.346590    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k" event={"ID":"95d7b734-0366-4e59-85ac-abbc5e0a18cd","Type":"ContainerStarted","Data":"65a9eb25ef60f16c752f2675525eff4ebf5461f61559b1ee6e68774ce2ee38c5"}
Apr 16 16:02:18.367075 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:18.367018    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k" podStartSLOduration=1.08436985 podStartE2EDuration="2.367001151s" podCreationTimestamp="2026-04-16 16:02:16 +0000 UTC" firstStartedPulling="2026-04-16 16:02:16.971397238 +0000 UTC m=+58.367481639" lastFinishedPulling="2026-04-16 16:02:18.254028541 +0000 UTC m=+59.650112940" observedRunningTime="2026-04-16 16:02:18.36665231 +0000 UTC m=+59.762736742" watchObservedRunningTime="2026-04-16 16:02:18.367001151 +0000 UTC m=+59.763085573"
Apr 16 16:02:19.277064 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.277033    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l8wbj"]
Apr 16 16:02:19.280076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.280060    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.283736 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.283710    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Apr 16 16:02:19.283821 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.283740    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Apr 16 16:02:19.284793 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.284776    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Apr 16 16:02:19.284912 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.284776    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Apr 16 16:02:19.284912 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.284779    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-frjrw\""
Apr 16 16:02:19.292388 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.292366    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8wbj"]
Apr 16 16:02:19.352585 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.352555    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerStarted","Data":"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"}
Apr 16 16:02:19.352876 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.352597    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerStarted","Data":"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"}
Apr 16 16:02:19.384552 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.384521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqp9m\" (UniqueName: \"kubernetes.io/projected/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-kube-api-access-tqp9m\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.384674 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.384566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-crio-socket\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.384674 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.384615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.384803 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.384686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-data-volume\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.384858 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.384836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486111 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486073    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tqp9m\" (UniqueName: \"kubernetes.io/projected/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-kube-api-access-tqp9m\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486120    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-crio-socket\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486152    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486178    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-data-volume\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486235    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-crio-socket\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486621    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-data-volume\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.486789 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.486773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.488639 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.488624    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.493746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.493715    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqp9m\" (UniqueName: \"kubernetes.io/projected/6371e86d-0f49-4ae0-93a2-7f9eed6363ba-kube-api-access-tqp9m\") pod \"insights-runtime-extractor-l8wbj\" (UID: \"6371e86d-0f49-4ae0-93a2-7f9eed6363ba\") " pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.590981 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.590885    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8wbj"
Apr 16 16:02:19.706739 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.706620    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8wbj"]
Apr 16 16:02:19.709306 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:19.709278    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6371e86d_0f49_4ae0_93a2_7f9eed6363ba.slice/crio-fa21d482b502adde5b9c723bf44bdd242d4213d48476fa6ea9846e0049b77b5b WatchSource:0}: Error finding container fa21d482b502adde5b9c723bf44bdd242d4213d48476fa6ea9846e0049b77b5b: Status 404 returned error can't find the container with id fa21d482b502adde5b9c723bf44bdd242d4213d48476fa6ea9846e0049b77b5b
Apr 16 16:02:19.857833 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.857752    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bf86bb564-mmcx5"]
Apr 16 16:02:19.863125 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.863099    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:19.865923 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.865904    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Apr 16 16:02:19.866248 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.866232    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Apr 16 16:02:19.866306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.866285    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Apr 16 16:02:19.866594 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.866577    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Apr 16 16:02:19.867161 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.867120    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Apr 16 16:02:19.867161 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.867149    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Apr 16 16:02:19.867343 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.867132    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Apr 16 16:02:19.867503 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.867482    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pp558\""
Apr 16 16:02:19.871407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.871389    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Apr 16 16:02:19.874194 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.874177    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf86bb564-mmcx5"]
Apr 16 16:02:19.990726 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.990693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-oauth-config\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:19.990726 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.990727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkhp\" (UniqueName: \"kubernetes.io/projected/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-kube-api-access-9pkhp\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:19.990932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.990807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-config\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:19.990932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.990839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-oauth-serving-cert\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:19.990932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.990863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-trusted-ca-bundle\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:19.990932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.990884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-service-ca\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:19.991056 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:19.990946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-serving-cert\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.091279 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.091242    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-serving-cert\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.091493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.091306    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-oauth-config\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.091493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.091332    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkhp\" (UniqueName: \"kubernetes.io/projected/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-kube-api-access-9pkhp\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.091493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.091386    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-config\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.091493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.091409    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-oauth-serving-cert\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.091715 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.091547    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-trusted-ca-bundle\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.091715 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.091606    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-service-ca\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.092191 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.092168    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-oauth-serving-cert\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.092308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.092204    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-config\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.092670 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.092638    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-service-ca\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.092956 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.092936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-trusted-ca-bundle\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.094037 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.094019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-serving-cert\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.094154 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.094102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-oauth-config\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.100774 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.100755    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkhp\" (UniqueName: \"kubernetes.io/projected/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-kube-api-access-9pkhp\") pod \"console-5bf86bb564-mmcx5\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") " pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.172867 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.172774    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:20.300934 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.300900    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf86bb564-mmcx5"]
Apr 16 16:02:20.361780 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.361745    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerStarted","Data":"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"}
Apr 16 16:02:20.363333 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.363305    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wbj" event={"ID":"6371e86d-0f49-4ae0-93a2-7f9eed6363ba","Type":"ContainerStarted","Data":"e1e5a2e019d0be418f6f97f7bcd47f72fd163c2f0954807ef8d7ce43d7423808"}
Apr 16 16:02:20.363486 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.363342    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wbj" event={"ID":"6371e86d-0f49-4ae0-93a2-7f9eed6363ba","Type":"ContainerStarted","Data":"fa21d482b502adde5b9c723bf44bdd242d4213d48476fa6ea9846e0049b77b5b"}
Apr 16 16:02:20.381801 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:20.381771    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc90c903_f9b8_4ca6_85dc_97939ec9a4f6.slice/crio-1368fcd9eb6df44b2c16cad55f2bf3739ed92dbd079508785c70bb5b32a1dc5c WatchSource:0}: Error finding container 1368fcd9eb6df44b2c16cad55f2bf3739ed92dbd079508785c70bb5b32a1dc5c: Status 404 returned error can't find the container with id 1368fcd9eb6df44b2c16cad55f2bf3739ed92dbd079508785c70bb5b32a1dc5c
Apr 16 16:02:20.386272 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:20.386219    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.625209736 podStartE2EDuration="7.386200618s" podCreationTimestamp="2026-04-16 16:02:13 +0000 UTC" firstStartedPulling="2026-04-16 16:02:14.52712846 +0000 UTC m=+55.923212858" lastFinishedPulling="2026-04-16 16:02:19.288119342 +0000 UTC m=+60.684203740" observedRunningTime="2026-04-16 16:02:20.385370554 +0000 UTC m=+61.781454974" watchObservedRunningTime="2026-04-16 16:02:20.386200618 +0000 UTC m=+61.782285039"
Apr 16 16:02:21.368228 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:21.368181    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf86bb564-mmcx5" event={"ID":"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6","Type":"ContainerStarted","Data":"1368fcd9eb6df44b2c16cad55f2bf3739ed92dbd079508785c70bb5b32a1dc5c"}
Apr 16 16:02:21.370222 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:21.370169    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wbj" event={"ID":"6371e86d-0f49-4ae0-93a2-7f9eed6363ba","Type":"ContainerStarted","Data":"5440682b27afcf6b3caedc21e79ee44d57429a580163e8f7ab11ab7304908e5f"}
Apr 16 16:02:24.381321 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.381280    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf86bb564-mmcx5" event={"ID":"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6","Type":"ContainerStarted","Data":"d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e"}
Apr 16 16:02:24.383201 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.383177    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8wbj" event={"ID":"6371e86d-0f49-4ae0-93a2-7f9eed6363ba","Type":"ContainerStarted","Data":"00077a0a1ea14f07545a9acf6f13ff8ea1252795be36a77be30a98ab8cb4933a"}
Apr 16 16:02:24.399044 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.399001    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5bf86bb564-mmcx5" podStartSLOduration=1.989788466 podStartE2EDuration="5.398989843s" podCreationTimestamp="2026-04-16 16:02:19 +0000 UTC" firstStartedPulling="2026-04-16 16:02:20.383513187 +0000 UTC m=+61.779597585" lastFinishedPulling="2026-04-16 16:02:23.792714561 +0000 UTC m=+65.188798962" observedRunningTime="2026-04-16 16:02:24.398009658 +0000 UTC m=+65.794094100" watchObservedRunningTime="2026-04-16 16:02:24.398989843 +0000 UTC m=+65.795074263"
Apr 16 16:02:24.415599 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.415553    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l8wbj" podStartSLOduration=1.398466438 podStartE2EDuration="5.41554156s" podCreationTimestamp="2026-04-16 16:02:19 +0000 UTC" firstStartedPulling="2026-04-16 16:02:19.768998728 +0000 UTC m=+61.165083132" lastFinishedPulling="2026-04-16 16:02:23.786073842 +0000 UTC m=+65.182158254" observedRunningTime="2026-04-16 16:02:24.415014055 +0000 UTC m=+65.811098475" watchObservedRunningTime="2026-04-16 16:02:24.41554156 +0000 UTC m=+65.811625980"
Apr 16 16:02:24.835002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.834949    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:02:24.835187 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.835025    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:02:24.838119 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.838093    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Apr 16 16:02:24.838981 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.838963    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Apr 16 16:02:24.847796 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.847768    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Apr 16 16:02:24.848074 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.848052    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53da8c88-a72c-4cf8-abde-f3a449cbdf3f-metrics-certs\") pod \"network-metrics-daemon-pbf54\" (UID: \"53da8c88-a72c-4cf8-abde-f3a449cbdf3f\") " pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:02:24.859002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:24.858976    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29nkz\" (UniqueName: \"kubernetes.io/projected/342d6e09-3c3b-470d-b383-b470e1c3a086-kube-api-access-29nkz\") pod \"network-check-target-8849t\" (UID: \"342d6e09-3c3b-470d-b383-b470e1c3a086\") " pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:02:25.122175 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.122096    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4vs75\""
Apr 16 16:02:25.128090 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.128069    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x4n9w\""
Apr 16 16:02:25.129909 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.129892    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:02:25.136654 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.136562    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbf54"
Apr 16 16:02:25.262152 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.262119    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8849t"]
Apr 16 16:02:25.265090 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:25.265063    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342d6e09_3c3b_470d_b383_b470e1c3a086.slice/crio-00f5c820f300f1ab69b3f516a1f8cddcece724dbea059d05e4b019996487b128 WatchSource:0}: Error finding container 00f5c820f300f1ab69b3f516a1f8cddcece724dbea059d05e4b019996487b128: Status 404 returned error can't find the container with id 00f5c820f300f1ab69b3f516a1f8cddcece724dbea059d05e4b019996487b128
Apr 16 16:02:25.280682 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.280658    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbf54"]
Apr 16 16:02:25.283641 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:02:25.283613    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53da8c88_a72c_4cf8_abde_f3a449cbdf3f.slice/crio-33b17e13494a0760bc396e6b64c730875e252ad9da475a19da204046d1fc6a3f WatchSource:0}: Error finding container 33b17e13494a0760bc396e6b64c730875e252ad9da475a19da204046d1fc6a3f: Status 404 returned error can't find the container with id 33b17e13494a0760bc396e6b64c730875e252ad9da475a19da204046d1fc6a3f
Apr 16 16:02:25.387394 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.387298    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbf54" event={"ID":"53da8c88-a72c-4cf8-abde-f3a449cbdf3f","Type":"ContainerStarted","Data":"33b17e13494a0760bc396e6b64c730875e252ad9da475a19da204046d1fc6a3f"}
Apr 16 16:02:25.388363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:25.388331    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8849t" event={"ID":"342d6e09-3c3b-470d-b383-b470e1c3a086","Type":"ContainerStarted","Data":"00f5c820f300f1ab69b3f516a1f8cddcece724dbea059d05e4b019996487b128"}
Apr 16 16:02:27.397121 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:27.397085    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbf54" event={"ID":"53da8c88-a72c-4cf8-abde-f3a449cbdf3f","Type":"ContainerStarted","Data":"9d05c1afbd77d2665eb023ea4f30ca71e7be4eb6842c177221342a7010217935"}
Apr 16 16:02:27.397121 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:27.397124    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbf54" event={"ID":"53da8c88-a72c-4cf8-abde-f3a449cbdf3f","Type":"ContainerStarted","Data":"7db64f31777e275de257287f3e6c13667117ecb8e52e46188bd345503f1487fb"}
Apr 16 16:02:27.414725 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:27.414350    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pbf54" podStartSLOduration=67.385013059 podStartE2EDuration="1m8.414331236s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:02:25.285380201 +0000 UTC m=+66.681464599" lastFinishedPulling="2026-04-16 16:02:26.314698372 +0000 UTC m=+67.710782776" observedRunningTime="2026-04-16 16:02:27.413203466 +0000 UTC m=+68.809287886" watchObservedRunningTime="2026-04-16 16:02:27.414331236 +0000 UTC m=+68.810415656"
Apr 16 16:02:28.401188 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:28.401098    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8849t" event={"ID":"342d6e09-3c3b-470d-b383-b470e1c3a086","Type":"ContainerStarted","Data":"0fcd184fc42e47f248e6b28016e049de8fa13968273794a2ba487f5aa5f430eb"}
Apr 16 16:02:28.401579 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:28.401271    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:02:28.415580 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:28.415533    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8849t" podStartSLOduration=66.618521808 podStartE2EDuration="1m9.415518861s" podCreationTimestamp="2026-04-16 16:01:19 +0000 UTC" firstStartedPulling="2026-04-16 16:02:25.267193102 +0000 UTC m=+66.663277500" lastFinishedPulling="2026-04-16 16:02:28.064190128 +0000 UTC m=+69.460274553" observedRunningTime="2026-04-16 16:02:28.414116582 +0000 UTC m=+69.810200999" watchObservedRunningTime="2026-04-16 16:02:28.415518861 +0000 UTC m=+69.811603278"
Apr 16 16:02:30.173191 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:30.173161    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:30.173623 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:30.173304    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:30.178075 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:30.178054    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:30.410815 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:30.410789    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:02:36.831380 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:36.831329    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:36.831380 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:36.831397    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:56.836092 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:56.836055    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:56.839929 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:56.839911    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55987d9d7d-8qs6k"
Apr 16 16:02:59.406547 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:02:59.406517    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8849t"
Apr 16 16:03:32.470923 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:32.470885    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:03:32.471542 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:32.471504    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="alertmanager" containerID="cri-o://f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f" gracePeriod=120
Apr 16 16:03:32.471683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:32.471514    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy" containerID="cri-o://6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841" gracePeriod=120
Apr 16 16:03:32.471683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:32.471562    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-web" containerID="cri-o://475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256" gracePeriod=120
Apr 16 16:03:32.471683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:32.471584    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-metric" containerID="cri-o://1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb" gracePeriod=120
Apr 16 16:03:32.471683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:32.471515    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="prom-label-proxy" containerID="cri-o://565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e" gracePeriod=120
Apr 16 16:03:32.471887 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:32.471728    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="config-reloader" containerID="cri-o://98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f" gracePeriod=120
Apr 16 16:03:33.582090 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582059    2578 generic.go:358] "Generic (PLEG): container finished" podID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerID="565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e" exitCode=0
Apr 16 16:03:33.582090 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582090    2578 generic.go:358] "Generic (PLEG): container finished" podID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerID="6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841" exitCode=0
Apr 16 16:03:33.582090 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582101    2578 generic.go:358] "Generic (PLEG): container finished" podID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerID="98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f" exitCode=0
Apr 16 16:03:33.582563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582111    2578 generic.go:358] "Generic (PLEG): container finished" podID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerID="f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f" exitCode=0
Apr 16 16:03:33.582563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582143    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"}
Apr 16 16:03:33.582563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582178    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"}
Apr 16 16:03:33.582563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582187    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"}
Apr 16 16:03:33.582563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.582207    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"}
Apr 16 16:03:33.703279 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.703256    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:33.801198 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801163    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-metrics-client-ca\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801358 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801204    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-web-config\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801358 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801234    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-trusted-ca-bundle\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801502 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801399    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqpqg\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-kube-api-access-nqpqg\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801502 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801434    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-main-db\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801613 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801545    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801613 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801577    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-config-out\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801613 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801594    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:03:33.801613 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801610    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-web\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801639    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-config-volume\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801654    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:03:33.801812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801675    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-tls-assets\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801723    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-cluster-tls-config\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.801812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801756    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:03:33.801812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801772    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.802124 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.801956    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls\") pod \"a9cd60df-ec8c-45db-8750-898a26ba4196\" (UID: \"a9cd60df-ec8c-45db-8750-898a26ba4196\") "
Apr 16 16:03:33.802176 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.802156    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-metrics-client-ca\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.802382 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.802176    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.802382 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.802193    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-alertmanager-main-db\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.804505 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.804471    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-kube-api-access-nqpqg" (OuterVolumeSpecName: "kube-api-access-nqpqg") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "kube-api-access-nqpqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:03:33.804804 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.804774    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:03:33.805252 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.805220    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:03:33.805252 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.805241    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:03:33.805372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.805262    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-config-volume" (OuterVolumeSpecName: "config-volume") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:03:33.805372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.805346    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:03:33.805802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.805774    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:03:33.806147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.806129    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-config-out" (OuterVolumeSpecName: "config-out") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:03:33.809106 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.809064    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:03:33.814573 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.814526    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-web-config" (OuterVolumeSpecName: "web-config") pod "a9cd60df-ec8c-45db-8750-898a26ba4196" (UID: "a9cd60df-ec8c-45db-8750-898a26ba4196"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:03:33.902878 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902853    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9cd60df-ec8c-45db-8750-898a26ba4196-config-out\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.902878 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902876    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902886    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-config-volume\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902915    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-tls-assets\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902923    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-cluster-tls-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902933    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902943    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-main-tls\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902953    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-web-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902962    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpqg\" (UniqueName: \"kubernetes.io/projected/a9cd60df-ec8c-45db-8750-898a26ba4196-kube-api-access-nqpqg\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:33.903009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:33.902970    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9cd60df-ec8c-45db-8750-898a26ba4196-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:03:34.587593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.587566    2578 generic.go:358] "Generic (PLEG): container finished" podID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerID="1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb" exitCode=0
Apr 16 16:03:34.587593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.587586    2578 generic.go:358] "Generic (PLEG): container finished" podID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerID="475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256" exitCode=0
Apr 16 16:03:34.587985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.587640    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"}
Apr 16 16:03:34.587985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.587681    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"}
Apr 16 16:03:34.587985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.587685    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.587985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.587693    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9cd60df-ec8c-45db-8750-898a26ba4196","Type":"ContainerDied","Data":"2cad4d3a524b937827c74bfba19adf6b81ce287414b3bfe545fca28552bf1751"}
Apr 16 16:03:34.587985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.587708    2578 scope.go:117] "RemoveContainer" containerID="565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"
Apr 16 16:03:34.595190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.595169    2578 scope.go:117] "RemoveContainer" containerID="1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"
Apr 16 16:03:34.601422 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.601409    2578 scope.go:117] "RemoveContainer" containerID="6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"
Apr 16 16:03:34.607438 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.607416    2578 scope.go:117] "RemoveContainer" containerID="475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"
Apr 16 16:03:34.611125 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.611105    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:03:34.614045 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.614023    2578 scope.go:117] "RemoveContainer" containerID="98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"
Apr 16 16:03:34.617684 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.617666    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:03:34.620588 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.620575    2578 scope.go:117] "RemoveContainer" containerID="f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"
Apr 16 16:03:34.626627 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.626611    2578 scope.go:117] "RemoveContainer" containerID="6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f"
Apr 16 16:03:34.632717 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.632703    2578 scope.go:117] "RemoveContainer" containerID="565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"
Apr 16 16:03:34.632957 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:03:34.632939    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e\": container with ID starting with 565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e not found: ID does not exist" containerID="565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"
Apr 16 16:03:34.633010 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.632965    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"} err="failed to get container status \"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e\": rpc error: code = NotFound desc = could not find container \"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e\": container with ID starting with 565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e not found: ID does not exist"
Apr 16 16:03:34.633010 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.632991    2578 scope.go:117] "RemoveContainer" containerID="1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"
Apr 16 16:03:34.633217 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:03:34.633203    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb\": container with ID starting with 1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb not found: ID does not exist" containerID="1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"
Apr 16 16:03:34.633262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633220    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"} err="failed to get container status \"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb\": rpc error: code = NotFound desc = could not find container \"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb\": container with ID starting with 1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb not found: ID does not exist"
Apr 16 16:03:34.633262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633235    2578 scope.go:117] "RemoveContainer" containerID="6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"
Apr 16 16:03:34.633490 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:03:34.633467    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841\": container with ID starting with 6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841 not found: ID does not exist" containerID="6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"
Apr 16 16:03:34.633559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633499    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"} err="failed to get container status \"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841\": rpc error: code = NotFound desc = could not find container \"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841\": container with ID starting with 6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841 not found: ID does not exist"
Apr 16 16:03:34.633559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633520    2578 scope.go:117] "RemoveContainer" containerID="475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"
Apr 16 16:03:34.633761 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:03:34.633744    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256\": container with ID starting with 475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256 not found: ID does not exist" containerID="475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"
Apr 16 16:03:34.633802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633767    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"} err="failed to get container status \"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256\": rpc error: code = NotFound desc = could not find container \"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256\": container with ID starting with 475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256 not found: ID does not exist"
Apr 16 16:03:34.633802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633782    2578 scope.go:117] "RemoveContainer" containerID="98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"
Apr 16 16:03:34.633982 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:03:34.633968    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f\": container with ID starting with 98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f not found: ID does not exist" containerID="98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"
Apr 16 16:03:34.634026 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633985    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"} err="failed to get container status \"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f\": rpc error: code = NotFound desc = could not find container \"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f\": container with ID starting with 98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f not found: ID does not exist"
Apr 16 16:03:34.634026 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.633998    2578 scope.go:117] "RemoveContainer" containerID="f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"
Apr 16 16:03:34.634187 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:03:34.634172    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f\": container with ID starting with f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f not found: ID does not exist" containerID="f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"
Apr 16 16:03:34.634226 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634192    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"} err="failed to get container status \"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f\": rpc error: code = NotFound desc = could not find container \"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f\": container with ID starting with f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f not found: ID does not exist"
Apr 16 16:03:34.634226 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634209    2578 scope.go:117] "RemoveContainer" containerID="6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f"
Apr 16 16:03:34.634465 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:03:34.634433    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f\": container with ID starting with 6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f not found: ID does not exist" containerID="6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f"
Apr 16 16:03:34.634543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f"} err="failed to get container status \"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f\": rpc error: code = NotFound desc = could not find container \"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f\": container with ID starting with 6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f not found: ID does not exist"
Apr 16 16:03:34.634543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634494    2578 scope.go:117] "RemoveContainer" containerID="565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"
Apr 16 16:03:34.634735 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634718    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e"} err="failed to get container status \"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e\": rpc error: code = NotFound desc = could not find container \"565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e\": container with ID starting with 565c8ef0f3b08ac0dea8eac591e32127981f1012e9629d475c0cac89860e691e not found: ID does not exist"
Apr 16 16:03:34.634798 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634736    2578 scope.go:117] "RemoveContainer" containerID="1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"
Apr 16 16:03:34.634965 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634950    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb"} err="failed to get container status \"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb\": rpc error: code = NotFound desc = could not find container \"1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb\": container with ID starting with 1959030cc931090a372aea2743d860e1ef2873abc8a975e004df9015818b14bb not found: ID does not exist"
Apr 16 16:03:34.635020 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.634965    2578 scope.go:117] "RemoveContainer" containerID="6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"
Apr 16 16:03:34.635194 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635169    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841"} err="failed to get container status \"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841\": rpc error: code = NotFound desc = could not find container \"6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841\": container with ID starting with 6aa15dc66c8b99deb1449c07563553cf8722a15d05a3a9cf4984a0f0eb1e0841 not found: ID does not exist"
Apr 16 16:03:34.635234 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635196    2578 scope.go:117] "RemoveContainer" containerID="475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"
Apr 16 16:03:34.635419 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635400    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256"} err="failed to get container status \"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256\": rpc error: code = NotFound desc = could not find container \"475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256\": container with ID starting with 475783f9f4fe0da52579da8b154dc5bb60e9fe175c0e460fe53bf1f0cdece256 not found: ID does not exist"
Apr 16 16:03:34.635495 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635420    2578 scope.go:117] "RemoveContainer" containerID="98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"
Apr 16 16:03:34.635658 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635640    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f"} err="failed to get container status \"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f\": rpc error: code = NotFound desc = could not find container \"98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f\": container with ID starting with 98426e77f00eba77b26afa159003aeb738e97b9efd6cf7e38a263c01f31b2e3f not found: ID does not exist"
Apr 16 16:03:34.635738 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635660    2578 scope.go:117] "RemoveContainer" containerID="f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"
Apr 16 16:03:34.635871 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635855    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f"} err="failed to get container status \"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f\": rpc error: code = NotFound desc = could not find container \"f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f\": container with ID starting with f855cd610a4bfeb3b0a7de53ae097ad01b96acb2694be7f197d0e6e402c8d80f not found: ID does not exist"
Apr 16 16:03:34.635910 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.635871    2578 scope.go:117] "RemoveContainer" containerID="6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f"
Apr 16 16:03:34.636057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.636041    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f"} err="failed to get container status \"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f\": rpc error: code = NotFound desc = could not find container \"6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f\": container with ID starting with 6a38be4dc5ec1afc87eaf27d278b6aac53676a7b2dcd10d6a16ce3ef6fdc146f not found: ID does not exist"
Apr 16 16:03:34.650392 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650373    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:03:34.650636 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650623    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="init-config-reloader"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650638    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="init-config-reloader"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650646    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="alertmanager"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650651    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="alertmanager"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650660    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650665    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650672    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="prom-label-proxy"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="prom-label-proxy"
Apr 16 16:03:34.650681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650683    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="config-reloader"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650687    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="config-reloader"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650697    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-web"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650702    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-web"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650709    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-metric"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650714    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-metric"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650750    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="prom-label-proxy"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650758    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650765    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-metric"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650771    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="config-reloader"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650778    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="alertmanager"
Apr 16 16:03:34.650895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.650784    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" containerName="kube-rbac-proxy-web"
Apr 16 16:03:34.655508 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.655493    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.659173 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.659154    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Apr 16 16:03:34.659318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.659193    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Apr 16 16:03:34.659318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.659159    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Apr 16 16:03:34.659318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.659163    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Apr 16 16:03:34.659571 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.659436    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5gsrq\""
Apr 16 16:03:34.659571 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.659465    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Apr 16 16:03:34.659859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.659842    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Apr 16 16:03:34.660268 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.660253    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Apr 16 16:03:34.660354 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.660318    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Apr 16 16:03:34.667036 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.667018    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Apr 16 16:03:34.674575 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.674552    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:03:34.809157 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae077b4d-8fc0-48f7-b89b-37d32a5db246-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f42vr\" (UniqueName: \"kubernetes.io/projected/ae077b4d-8fc0-48f7-b89b-37d32a5db246-kube-api-access-f42vr\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae077b4d-8fc0-48f7-b89b-37d32a5db246-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae077b4d-8fc0-48f7-b89b-37d32a5db246-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-web-config\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae077b4d-8fc0-48f7-b89b-37d32a5db246-config-out\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-config-volume\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae077b4d-8fc0-48f7-b89b-37d32a5db246-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.809553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.809495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910032 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.909970    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae077b4d-8fc0-48f7-b89b-37d32a5db246-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910032 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910001    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae077b4d-8fc0-48f7-b89b-37d32a5db246-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910032 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910021    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-web-config\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910218 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910044    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae077b4d-8fc0-48f7-b89b-37d32a5db246-config-out\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910218 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910075    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910302 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910259    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-config-volume\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910302 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910294    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910398 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910324    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae077b4d-8fc0-48f7-b89b-37d32a5db246-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910398 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910355    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910427    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910474    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae077b4d-8fc0-48f7-b89b-37d32a5db246-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910512    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910535    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f42vr\" (UniqueName: \"kubernetes.io/projected/ae077b4d-8fc0-48f7-b89b-37d32a5db246-kube-api-access-f42vr\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.910848 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.910819    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ae077b4d-8fc0-48f7-b89b-37d32a5db246-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.911301 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.911275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ae077b4d-8fc0-48f7-b89b-37d32a5db246-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.912100 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.912070    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae077b4d-8fc0-48f7-b89b-37d32a5db246-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.913391 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.913360    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ae077b4d-8fc0-48f7-b89b-37d32a5db246-config-out\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.913537 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.913509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-config-volume\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.913734 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.913702    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-web-config\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.913843 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.913816    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.913905 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.913858    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ae077b4d-8fc0-48f7-b89b-37d32a5db246-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.914229 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.914206    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.914315 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.914275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.914720 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.914700    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.915619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.915598    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ae077b4d-8fc0-48f7-b89b-37d32a5db246-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.923478 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.923435    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f42vr\" (UniqueName: \"kubernetes.io/projected/ae077b4d-8fc0-48f7-b89b-37d32a5db246-kube-api-access-f42vr\") pod \"alertmanager-main-0\" (UID: \"ae077b4d-8fc0-48f7-b89b-37d32a5db246\") " pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:34.964859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:34.964840    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Apr 16 16:03:35.091805 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:35.091720    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Apr 16 16:03:35.093924 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:03:35.093898    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae077b4d_8fc0_48f7_b89b_37d32a5db246.slice/crio-0bb5ced45fd246e7cc56213960775eac3d143dbad0153944b67d8f4ca0593675 WatchSource:0}: Error finding container 0bb5ced45fd246e7cc56213960775eac3d143dbad0153944b67d8f4ca0593675: Status 404 returned error can't find the container with id 0bb5ced45fd246e7cc56213960775eac3d143dbad0153944b67d8f4ca0593675
Apr 16 16:03:35.106760 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:35.106737    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9cd60df-ec8c-45db-8750-898a26ba4196" path="/var/lib/kubelet/pods/a9cd60df-ec8c-45db-8750-898a26ba4196/volumes"
Apr 16 16:03:35.591529 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:35.591502    2578 generic.go:358] "Generic (PLEG): container finished" podID="ae077b4d-8fc0-48f7-b89b-37d32a5db246" containerID="f0b1f1b715b8bd6f8bd66065611639d287c2ef450d319433ecb25b2c003144f3" exitCode=0
Apr 16 16:03:35.591953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:35.591598    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerDied","Data":"f0b1f1b715b8bd6f8bd66065611639d287c2ef450d319433ecb25b2c003144f3"}
Apr 16 16:03:35.591953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:35.591646    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerStarted","Data":"0bb5ced45fd246e7cc56213960775eac3d143dbad0153944b67d8f4ca0593675"}
Apr 16 16:03:36.514489 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.514433    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55c89b586b-trw2s"]
Apr 16 16:03:36.518024 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.518009    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522258 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522237    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Apr 16 16:03:36.522258 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522254    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Apr 16 16:03:36.522431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522242    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Apr 16 16:03:36.522635 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6b7j\" (UniqueName: \"kubernetes.io/projected/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-kube-api-access-c6b7j\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-secret-telemeter-client\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-metrics-client-ca\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522780 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-federate-client-tls\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522780 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-telemeter-client-tls\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522780 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522920 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-serving-certs-ca-bundle\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.522920 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.522859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.523346 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.523320    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-txg4f\""
Apr 16 16:03:36.523838 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.523823    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Apr 16 16:03:36.524275 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.524095    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Apr 16 16:03:36.528363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.528344    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Apr 16 16:03:36.538207 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.538177    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55c89b586b-trw2s"]
Apr 16 16:03:36.602903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.602870    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerStarted","Data":"e0c2d0dae4c9bbbb0d6f242d55bcb83279bd3dda78ee6e48906d7a6f4f44288b"}
Apr 16 16:03:36.603343 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.603323    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerStarted","Data":"063e7f7d8729bf2b975f63251fd0627ae857dbbe1a9d9bfad7a1c0334f03d8d4"}
Apr 16 16:03:36.603500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.603485    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerStarted","Data":"8a1b09d49ad6d765bd7382245925cdd6283107a59cb2660db7be8b881f8062a4"}
Apr 16 16:03:36.603602 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.603589    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerStarted","Data":"5ca60320f34b86baec3d520640cb21c32e31d3bbef179b0ee0141c7fbcb3a463"}
Apr 16 16:03:36.603710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.603697    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerStarted","Data":"5013aacc7489d113c97a2dfe4ac103f94c48317d0fdcecde083f4b7386250b53"}
Apr 16 16:03:36.603810 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.603798    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ae077b4d-8fc0-48f7-b89b-37d32a5db246","Type":"ContainerStarted","Data":"34c1760d2fd1f74220b73c1476ce0126db55dd6521e30f1a2556594392a78e86"}
Apr 16 16:03:36.624514 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624487    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.624655 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624590    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-serving-certs-ca-bundle\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.624655 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624618    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.624776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624683    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c6b7j\" (UniqueName: \"kubernetes.io/projected/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-kube-api-access-c6b7j\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.624776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624735    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-secret-telemeter-client\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.624776 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624759    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-metrics-client-ca\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.624953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624819    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-federate-client-tls\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.624953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.624855    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-telemeter-client-tls\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.625958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.625933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-serving-certs-ca-bundle\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.626058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.625991    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-metrics-client-ca\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.626058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.625933    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.628561 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.628531    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-federate-client-tls\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.628976 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.628937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.629543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.629524    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-telemeter-client-tls\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.630259 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.630233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-secret-telemeter-client\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.634374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.634331    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.6343198660000002 podStartE2EDuration="2.634319866s" podCreationTimestamp="2026-04-16 16:03:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:03:36.632236662 +0000 UTC m=+138.028321082" watchObservedRunningTime="2026-04-16 16:03:36.634319866 +0000 UTC m=+138.030404286"
Apr 16 16:03:36.636614 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.636574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6b7j\" (UniqueName: \"kubernetes.io/projected/177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02-kube-api-access-c6b7j\") pod \"telemeter-client-55c89b586b-trw2s\" (UID: \"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02\") " pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.828131 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.828035    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s"
Apr 16 16:03:36.975500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:36.975470    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55c89b586b-trw2s"]
Apr 16 16:03:36.977093 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:03:36.977063    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177aa2ad_9c6e_40a6_8aa4_b2dd06fdbf02.slice/crio-908d0d344f95d18b5b08bd38e629cad69367402d53781907cfdd000e84280dc8 WatchSource:0}: Error finding container 908d0d344f95d18b5b08bd38e629cad69367402d53781907cfdd000e84280dc8: Status 404 returned error can't find the container with id 908d0d344f95d18b5b08bd38e629cad69367402d53781907cfdd000e84280dc8
Apr 16 16:03:37.608142 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:37.608108    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s" event={"ID":"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02","Type":"ContainerStarted","Data":"908d0d344f95d18b5b08bd38e629cad69367402d53781907cfdd000e84280dc8"}
Apr 16 16:03:39.615893 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:39.615813    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s" event={"ID":"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02","Type":"ContainerStarted","Data":"621dc06d2e40f9cd60d31a943c140f08905f44b4e82d3c37ef9816c94c1b5bcc"}
Apr 16 16:03:39.615893 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:39.615860    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s" event={"ID":"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02","Type":"ContainerStarted","Data":"a44d1f3793dfa1cbee86634c6aa016f6fa2197d4c71a84cbaee3d924d96f604b"}
Apr 16 16:03:39.615893 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:39.615875    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s" event={"ID":"177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02","Type":"ContainerStarted","Data":"651f5334e58e897627b81f2fbfa24cdd5e39e635de7f3335ff3b094226c63bff"}
Apr 16 16:03:39.640368 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:39.640326    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55c89b586b-trw2s" podStartSLOduration=1.255593102 podStartE2EDuration="3.640312322s" podCreationTimestamp="2026-04-16 16:03:36 +0000 UTC" firstStartedPulling="2026-04-16 16:03:36.978769833 +0000 UTC m=+138.374854231" lastFinishedPulling="2026-04-16 16:03:39.363489038 +0000 UTC m=+140.759573451" observedRunningTime="2026-04-16 16:03:39.638946954 +0000 UTC m=+141.035031373" watchObservedRunningTime="2026-04-16 16:03:39.640312322 +0000 UTC m=+141.036396741"
Apr 16 16:03:40.299123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.299094    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-9f55f9cb9-w8jm5"]
Apr 16 16:03:40.302411 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.302392    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.329780 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.329746    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f55f9cb9-w8jm5"]
Apr 16 16:03:40.353079 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.353060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsvxj\" (UniqueName: \"kubernetes.io/projected/987d4e67-8c7c-4a14-99e8-760b254103df-kube-api-access-gsvxj\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.353183 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.353091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-oauth-config\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.353183 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.353109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-service-ca\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.353183 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.353132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-oauth-serving-cert\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.353286 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.353196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-serving-cert\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.353286 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.353222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-trusted-ca-bundle\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.353286 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.353261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-console-config\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.454522 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.454502    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-serving-cert\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.454522 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.454528    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-trusted-ca-bundle\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.454690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.454561    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-console-config\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.454690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.454584    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gsvxj\" (UniqueName: \"kubernetes.io/projected/987d4e67-8c7c-4a14-99e8-760b254103df-kube-api-access-gsvxj\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.454690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.454618    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-oauth-config\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.454690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.454634    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-service-ca\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.454690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.454649    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-oauth-serving-cert\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.455347 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.455290    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-console-config\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.455464 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.455348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-oauth-serving-cert\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.455464 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.455361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-service-ca\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.456960 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.456938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-trusted-ca-bundle\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.457162 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.457141    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-serving-cert\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.457206 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.457163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-oauth-config\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.464019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.463994    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsvxj\" (UniqueName: \"kubernetes.io/projected/987d4e67-8c7c-4a14-99e8-760b254103df-kube-api-access-gsvxj\") pod \"console-9f55f9cb9-w8jm5\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") " pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.612366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.612277    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:40.757274 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:40.757238    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9f55f9cb9-w8jm5"]
Apr 16 16:03:41.627958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:41.627911    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f55f9cb9-w8jm5" event={"ID":"987d4e67-8c7c-4a14-99e8-760b254103df","Type":"ContainerStarted","Data":"3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1"}
Apr 16 16:03:41.627958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:41.627951    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f55f9cb9-w8jm5" event={"ID":"987d4e67-8c7c-4a14-99e8-760b254103df","Type":"ContainerStarted","Data":"a3b8b6b77005c21bd1429dc943060ccb2b03dd39830fd92ff349fb11f4d92506"}
Apr 16 16:03:41.653221 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:41.653174    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-9f55f9cb9-w8jm5" podStartSLOduration=1.653157464 podStartE2EDuration="1.653157464s" podCreationTimestamp="2026-04-16 16:03:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:03:41.652242999 +0000 UTC m=+143.048327441" watchObservedRunningTime="2026-04-16 16:03:41.653157464 +0000 UTC m=+143.049241884"
Apr 16 16:03:50.612591 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:50.612543    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:50.612591 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:50.612596    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:50.617437 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:50.617412    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:50.667313 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:50.667284    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:03:50.722634 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:03:50.722603    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf86bb564-mmcx5"]
Apr 16 16:04:15.747435 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:15.747374    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5bf86bb564-mmcx5" podUID="dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" containerName="console" containerID="cri-o://d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e" gracePeriod=15
Apr 16 16:04:15.978889 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:15.978865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bf86bb564-mmcx5_dc90c903-f9b8-4ca6-85dc-97939ec9a4f6/console/0.log"
Apr 16 16:04:15.979013 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:15.978938    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:04:16.099071 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099012    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-oauth-serving-cert\") pod \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") "
Apr 16 16:04:16.099071 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099040    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-oauth-config\") pod \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") "
Apr 16 16:04:16.099260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099076    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-trusted-ca-bundle\") pod \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") "
Apr 16 16:04:16.099260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099097    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-service-ca\") pod \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") "
Apr 16 16:04:16.099260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099122    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-serving-cert\") pod \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") "
Apr 16 16:04:16.099260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099159    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-config\") pod \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") "
Apr 16 16:04:16.099260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099215    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pkhp\" (UniqueName: \"kubernetes.io/projected/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-kube-api-access-9pkhp\") pod \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\" (UID: \"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6\") "
Apr 16 16:04:16.099622 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099587    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-service-ca" (OuterVolumeSpecName: "service-ca") pod "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" (UID: "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:04:16.099707 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099616    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" (UID: "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:04:16.099707 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099624    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" (UID: "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:04:16.099707 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.099669    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-config" (OuterVolumeSpecName: "console-config") pod "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" (UID: "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:04:16.101411 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.101383    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" (UID: "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:04:16.101411 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.101398    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" (UID: "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:04:16.101595 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.101579    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-kube-api-access-9pkhp" (OuterVolumeSpecName: "kube-api-access-9pkhp") pod "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" (UID: "dc90c903-f9b8-4ca6-85dc-97939ec9a4f6"). InnerVolumeSpecName "kube-api-access-9pkhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:04:16.199979 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.199955    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-trusted-ca-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:04:16.199979 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.199974    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-service-ca\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:04:16.200101 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.199985    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:04:16.200101 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.199994    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:04:16.200101 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.200004    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkhp\" (UniqueName: \"kubernetes.io/projected/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-kube-api-access-9pkhp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:04:16.200101 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.200012    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-oauth-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:04:16.200101 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.200021    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6-console-oauth-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:04:16.739289 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.739259    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bf86bb564-mmcx5_dc90c903-f9b8-4ca6-85dc-97939ec9a4f6/console/0.log"
Apr 16 16:04:16.739480 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.739305    2578 generic.go:358] "Generic (PLEG): container finished" podID="dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" containerID="d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e" exitCode=2
Apr 16 16:04:16.739480 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.739353    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf86bb564-mmcx5" event={"ID":"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6","Type":"ContainerDied","Data":"d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e"}
Apr 16 16:04:16.739480 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.739378    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf86bb564-mmcx5" event={"ID":"dc90c903-f9b8-4ca6-85dc-97939ec9a4f6","Type":"ContainerDied","Data":"1368fcd9eb6df44b2c16cad55f2bf3739ed92dbd079508785c70bb5b32a1dc5c"}
Apr 16 16:04:16.739480 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.739406    2578 scope.go:117] "RemoveContainer" containerID="d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e"
Apr 16 16:04:16.739633 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.739410    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf86bb564-mmcx5"
Apr 16 16:04:16.748751 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.748574    2578 scope.go:117] "RemoveContainer" containerID="d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e"
Apr 16 16:04:16.748985 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:04:16.748864    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e\": container with ID starting with d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e not found: ID does not exist" containerID="d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e"
Apr 16 16:04:16.748985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.748884    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e"} err="failed to get container status \"d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e\": rpc error: code = NotFound desc = could not find container \"d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e\": container with ID starting with d3561290a21edc86ef1fda7b12a8d3add25503f69490a31e9888099289857f4e not found: ID does not exist"
Apr 16 16:04:16.762294 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.762266    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf86bb564-mmcx5"]
Apr 16 16:04:16.769269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:16.769245    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bf86bb564-mmcx5"]
Apr 16 16:04:17.106932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:17.106872    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" path="/var/lib/kubelet/pods/dc90c903-f9b8-4ca6-85dc-97939ec9a4f6/volumes"
Apr 16 16:04:59.933664 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.933626    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-56b4dccb4d-gsqx4"]
Apr 16 16:04:59.934140 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.933897    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" containerName="console"
Apr 16 16:04:59.934140 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.933909    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" containerName="console"
Apr 16 16:04:59.934140 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.933964    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="dc90c903-f9b8-4ca6-85dc-97939ec9a4f6" containerName="console"
Apr 16 16:04:59.936950 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.936934    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:04:59.950053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.950023    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4dccb4d-gsqx4"]
Apr 16 16:04:59.996070 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.996040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-trusted-ca-bundle\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:04:59.996070 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.996071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62mc\" (UniqueName: \"kubernetes.io/projected/9cd41e70-18c1-4069-a0a1-c4c318bc942a-kube-api-access-q62mc\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:04:59.996305 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.996097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-oauth-serving-cert\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:04:59.996305 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.996209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-oauth-config\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:04:59.996305 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.996260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-config\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:04:59.996305 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.996299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-serving-cert\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:04:59.996523 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:04:59.996327    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-service-ca\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.097728 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.097699    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-config\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.097728 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.097729    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-serving-cert\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.097922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.097746    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-service-ca\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.097922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.097789    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-trusted-ca-bundle\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.097922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.097815    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q62mc\" (UniqueName: \"kubernetes.io/projected/9cd41e70-18c1-4069-a0a1-c4c318bc942a-kube-api-access-q62mc\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.097922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.097847    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-oauth-serving-cert\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.097922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.097892    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-oauth-config\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.098525 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.098497    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-config\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.098635 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.098533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-service-ca\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.098694 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.098659    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-oauth-serving-cert\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.098857 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.098835    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-trusted-ca-bundle\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.100380 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.100357    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-oauth-config\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.100540 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.100522    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-serving-cert\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.105882 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.105863    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62mc\" (UniqueName: \"kubernetes.io/projected/9cd41e70-18c1-4069-a0a1-c4c318bc942a-kube-api-access-q62mc\") pod \"console-56b4dccb4d-gsqx4\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") " pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.245549 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.245507    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:00.376677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.376648    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b4dccb4d-gsqx4"]
Apr 16 16:05:00.380364 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:05:00.380336    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd41e70_18c1_4069_a0a1_c4c318bc942a.slice/crio-0a1abaad544de4f6a59360c83c651c4527f9e84383fbce2c269065d81a493335 WatchSource:0}: Error finding container 0a1abaad544de4f6a59360c83c651c4527f9e84383fbce2c269065d81a493335: Status 404 returned error can't find the container with id 0a1abaad544de4f6a59360c83c651c4527f9e84383fbce2c269065d81a493335
Apr 16 16:05:00.872742 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.872703    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4dccb4d-gsqx4" event={"ID":"9cd41e70-18c1-4069-a0a1-c4c318bc942a","Type":"ContainerStarted","Data":"1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7"}
Apr 16 16:05:00.872742 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.872743    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4dccb4d-gsqx4" event={"ID":"9cd41e70-18c1-4069-a0a1-c4c318bc942a","Type":"ContainerStarted","Data":"0a1abaad544de4f6a59360c83c651c4527f9e84383fbce2c269065d81a493335"}
Apr 16 16:05:00.891160 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:00.891104    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-56b4dccb4d-gsqx4" podStartSLOduration=1.891088407 podStartE2EDuration="1.891088407s" podCreationTimestamp="2026-04-16 16:04:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:05:00.890339352 +0000 UTC m=+222.286423777" watchObservedRunningTime="2026-04-16 16:05:00.891088407 +0000 UTC m=+222.287172829"
Apr 16 16:05:10.246420 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:10.246386    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:10.246420 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:10.246428    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:10.250782 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:10.250759    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:10.903595 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:10.903569    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:05:10.956468 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:10.956412    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f55f9cb9-w8jm5"]
Apr 16 16:05:29.213505 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.213426    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mjzrr"]
Apr 16 16:05:29.216424 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.216410    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.220302 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.220283    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Apr 16 16:05:29.235142 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.235121    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mjzrr"]
Apr 16 16:05:29.304627 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.304597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f822bab8-379f-4f6a-bffd-cbfca5905808-original-pull-secret\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.304771 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.304637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f822bab8-379f-4f6a-bffd-cbfca5905808-dbus\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.304771 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.304715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f822bab8-379f-4f6a-bffd-cbfca5905808-kubelet-config\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.405047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.405021    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f822bab8-379f-4f6a-bffd-cbfca5905808-original-pull-secret\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.405172 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.405055    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f822bab8-379f-4f6a-bffd-cbfca5905808-dbus\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.405237 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.405194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f822bab8-379f-4f6a-bffd-cbfca5905808-dbus\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.405237 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.405196    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f822bab8-379f-4f6a-bffd-cbfca5905808-kubelet-config\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.405314 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.405270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f822bab8-379f-4f6a-bffd-cbfca5905808-kubelet-config\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.407372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.407348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f822bab8-379f-4f6a-bffd-cbfca5905808-original-pull-secret\") pod \"global-pull-secret-syncer-mjzrr\" (UID: \"f822bab8-379f-4f6a-bffd-cbfca5905808\") " pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.525019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.524967    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mjzrr"
Apr 16 16:05:29.653393 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.653358    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mjzrr"]
Apr 16 16:05:29.655714 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:05:29.655690    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf822bab8_379f_4f6a_bffd_cbfca5905808.slice/crio-7976eb45ca09cf831a3bf32edbb24c5678cbe99471b91869b2db4ca9142c7591 WatchSource:0}: Error finding container 7976eb45ca09cf831a3bf32edbb24c5678cbe99471b91869b2db4ca9142c7591: Status 404 returned error can't find the container with id 7976eb45ca09cf831a3bf32edbb24c5678cbe99471b91869b2db4ca9142c7591
Apr 16 16:05:29.954783 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:29.954702    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mjzrr" event={"ID":"f822bab8-379f-4f6a-bffd-cbfca5905808","Type":"ContainerStarted","Data":"7976eb45ca09cf831a3bf32edbb24c5678cbe99471b91869b2db4ca9142c7591"}
Apr 16 16:05:33.967596 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:33.967554    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mjzrr" event={"ID":"f822bab8-379f-4f6a-bffd-cbfca5905808","Type":"ContainerStarted","Data":"f287ef42d1ba59182eea2b271008be0b6eb8e08a31e705b36ba27f084114e63b"}
Apr 16 16:05:33.984416 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:33.984364    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mjzrr" podStartSLOduration=1.283792555 podStartE2EDuration="4.984350189s" podCreationTimestamp="2026-04-16 16:05:29 +0000 UTC" firstStartedPulling="2026-04-16 16:05:29.657307268 +0000 UTC m=+251.053391666" lastFinishedPulling="2026-04-16 16:05:33.357864902 +0000 UTC m=+254.753949300" observedRunningTime="2026-04-16 16:05:33.983648969 +0000 UTC m=+255.379733391" watchObservedRunningTime="2026-04-16 16:05:33.984350189 +0000 UTC m=+255.380434609"
Apr 16 16:05:35.975455 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:35.975399    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-9f55f9cb9-w8jm5" podUID="987d4e67-8c7c-4a14-99e8-760b254103df" containerName="console" containerID="cri-o://3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1" gracePeriod=15
Apr 16 16:05:36.208287 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.208262    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9f55f9cb9-w8jm5_987d4e67-8c7c-4a14-99e8-760b254103df/console/0.log"
Apr 16 16:05:36.208392 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.208325    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:05:36.261724 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.261663    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsvxj\" (UniqueName: \"kubernetes.io/projected/987d4e67-8c7c-4a14-99e8-760b254103df-kube-api-access-gsvxj\") pod \"987d4e67-8c7c-4a14-99e8-760b254103df\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") "
Apr 16 16:05:36.261724 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.261706    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-serving-cert\") pod \"987d4e67-8c7c-4a14-99e8-760b254103df\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") "
Apr 16 16:05:36.261886 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.261739    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-oauth-config\") pod \"987d4e67-8c7c-4a14-99e8-760b254103df\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") "
Apr 16 16:05:36.261942 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.261888    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-console-config\") pod \"987d4e67-8c7c-4a14-99e8-760b254103df\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") "
Apr 16 16:05:36.262000 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.261936    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-oauth-serving-cert\") pod \"987d4e67-8c7c-4a14-99e8-760b254103df\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") "
Apr 16 16:05:36.262000 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.261989    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-trusted-ca-bundle\") pod \"987d4e67-8c7c-4a14-99e8-760b254103df\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") "
Apr 16 16:05:36.262096 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.262023    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-service-ca\") pod \"987d4e67-8c7c-4a14-99e8-760b254103df\" (UID: \"987d4e67-8c7c-4a14-99e8-760b254103df\") "
Apr 16 16:05:36.262580 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.262354    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-console-config" (OuterVolumeSpecName: "console-config") pod "987d4e67-8c7c-4a14-99e8-760b254103df" (UID: "987d4e67-8c7c-4a14-99e8-760b254103df"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:05:36.262580 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.262395    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "987d4e67-8c7c-4a14-99e8-760b254103df" (UID: "987d4e67-8c7c-4a14-99e8-760b254103df"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:05:36.262580 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.262541    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-service-ca" (OuterVolumeSpecName: "service-ca") pod "987d4e67-8c7c-4a14-99e8-760b254103df" (UID: "987d4e67-8c7c-4a14-99e8-760b254103df"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:05:36.262580 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.262559    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "987d4e67-8c7c-4a14-99e8-760b254103df" (UID: "987d4e67-8c7c-4a14-99e8-760b254103df"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:05:36.263948 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.263930    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "987d4e67-8c7c-4a14-99e8-760b254103df" (UID: "987d4e67-8c7c-4a14-99e8-760b254103df"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:05:36.264426 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.264409    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "987d4e67-8c7c-4a14-99e8-760b254103df" (UID: "987d4e67-8c7c-4a14-99e8-760b254103df"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:05:36.264528 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.264466    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987d4e67-8c7c-4a14-99e8-760b254103df-kube-api-access-gsvxj" (OuterVolumeSpecName: "kube-api-access-gsvxj") pod "987d4e67-8c7c-4a14-99e8-760b254103df" (UID: "987d4e67-8c7c-4a14-99e8-760b254103df"). InnerVolumeSpecName "kube-api-access-gsvxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:05:36.363386 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.363359    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsvxj\" (UniqueName: \"kubernetes.io/projected/987d4e67-8c7c-4a14-99e8-760b254103df-kube-api-access-gsvxj\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:05:36.363386 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.363382    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:05:36.363386 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.363392    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/987d4e67-8c7c-4a14-99e8-760b254103df-console-oauth-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:05:36.363593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.363403    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-console-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:05:36.363593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.363411    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-oauth-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:05:36.363593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.363420    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-trusted-ca-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:05:36.363593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.363429    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/987d4e67-8c7c-4a14-99e8-760b254103df-service-ca\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:05:36.978334 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.978310    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9f55f9cb9-w8jm5_987d4e67-8c7c-4a14-99e8-760b254103df/console/0.log"
Apr 16 16:05:36.978736 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.978347    2578 generic.go:358] "Generic (PLEG): container finished" podID="987d4e67-8c7c-4a14-99e8-760b254103df" containerID="3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1" exitCode=2
Apr 16 16:05:36.978736 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.978415    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f55f9cb9-w8jm5" event={"ID":"987d4e67-8c7c-4a14-99e8-760b254103df","Type":"ContainerDied","Data":"3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1"}
Apr 16 16:05:36.978736 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.978422    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9f55f9cb9-w8jm5"
Apr 16 16:05:36.978736 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.978437    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9f55f9cb9-w8jm5" event={"ID":"987d4e67-8c7c-4a14-99e8-760b254103df","Type":"ContainerDied","Data":"a3b8b6b77005c21bd1429dc943060ccb2b03dd39830fd92ff349fb11f4d92506"}
Apr 16 16:05:36.978736 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.978469    2578 scope.go:117] "RemoveContainer" containerID="3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1"
Apr 16 16:05:36.986441 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.986425    2578 scope.go:117] "RemoveContainer" containerID="3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1"
Apr 16 16:05:36.986755 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:05:36.986734    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1\": container with ID starting with 3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1 not found: ID does not exist" containerID="3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1"
Apr 16 16:05:36.986808 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:36.986765    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1"} err="failed to get container status \"3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1\": rpc error: code = NotFound desc = could not find container \"3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1\": container with ID starting with 3b70dc9e003affe6102b2fde0cd805cd64692f7b5f5e82f98e3750dc9a84e8e1 not found: ID does not exist"
Apr 16 16:05:37.004131 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:37.004108    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9f55f9cb9-w8jm5"]
Apr 16 16:05:37.008736 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:37.008714    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9f55f9cb9-w8jm5"]
Apr 16 16:05:37.107873 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:05:37.107847    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="987d4e67-8c7c-4a14-99e8-760b254103df" path="/var/lib/kubelet/pods/987d4e67-8c7c-4a14-99e8-760b254103df/volumes"
Apr 16 16:06:19.021143 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:06:19.021113    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:06:19.023381 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:06:19.023360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:06:19.024369 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:06:19.024336    2578 kubelet.go:1628] "Image garbage collection succeeded"
Apr 16 16:08:42.563599 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.563561    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db74f6b9d-2qz9p"]
Apr 16 16:08:42.564117 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.563887    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="987d4e67-8c7c-4a14-99e8-760b254103df" containerName="console"
Apr 16 16:08:42.564117 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.563900    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="987d4e67-8c7c-4a14-99e8-760b254103df" containerName="console"
Apr 16 16:08:42.564117 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.563966    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="987d4e67-8c7c-4a14-99e8-760b254103df" containerName="console"
Apr 16 16:08:42.566779 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.566761    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.586792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.586756    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db74f6b9d-2qz9p"]
Apr 16 16:08:42.605688 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.605655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-oauth-serving-cert\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.605873 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.605722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-trusted-ca-bundle\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.605873 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.605769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-console-config\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.605873 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.605788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5461e083-38c9-41c5-9e80-3fb2f3603562-console-serving-cert\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.605873 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.605820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2r9\" (UniqueName: \"kubernetes.io/projected/5461e083-38c9-41c5-9e80-3fb2f3603562-kube-api-access-np2r9\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.606003 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.605915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-service-ca\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.606003 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.605955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5461e083-38c9-41c5-9e80-3fb2f3603562-console-oauth-config\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.706694 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.706658    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-service-ca\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.706917 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.706716    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5461e083-38c9-41c5-9e80-3fb2f3603562-console-oauth-config\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.706917 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.706743    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-oauth-serving-cert\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.706917 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.706870    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-trusted-ca-bundle\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.707058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.706928    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-console-config\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.707058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.706955    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5461e083-38c9-41c5-9e80-3fb2f3603562-console-serving-cert\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.707058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.707002    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-np2r9\" (UniqueName: \"kubernetes.io/projected/5461e083-38c9-41c5-9e80-3fb2f3603562-kube-api-access-np2r9\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.707587 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.707558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-oauth-serving-cert\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.707909 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.707605    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-service-ca\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.707909 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.707796    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-trusted-ca-bundle\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.708053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.707966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5461e083-38c9-41c5-9e80-3fb2f3603562-console-config\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.709621 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.709599    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5461e083-38c9-41c5-9e80-3fb2f3603562-console-oauth-config\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.709831 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.709810    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5461e083-38c9-41c5-9e80-3fb2f3603562-console-serving-cert\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.719704 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.719678    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2r9\" (UniqueName: \"kubernetes.io/projected/5461e083-38c9-41c5-9e80-3fb2f3603562-kube-api-access-np2r9\") pod \"console-5db74f6b9d-2qz9p\" (UID: \"5461e083-38c9-41c5-9e80-3fb2f3603562\") " pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:42.876597 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:42.876490    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:43.005500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:43.005308    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db74f6b9d-2qz9p"]
Apr 16 16:08:43.009873 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:08:43.009833    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5461e083_38c9_41c5_9e80_3fb2f3603562.slice/crio-504435ed88a66bab8e2ac41db182ad0a78adcf073cdff8aeefa1ffb449ad1e81 WatchSource:0}: Error finding container 504435ed88a66bab8e2ac41db182ad0a78adcf073cdff8aeefa1ffb449ad1e81: Status 404 returned error can't find the container with id 504435ed88a66bab8e2ac41db182ad0a78adcf073cdff8aeefa1ffb449ad1e81
Apr 16 16:08:43.011810 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:43.011788    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:08:43.488817 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:43.488771    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db74f6b9d-2qz9p" event={"ID":"5461e083-38c9-41c5-9e80-3fb2f3603562","Type":"ContainerStarted","Data":"323c9145bcbe482139c4df0f72c0d1b32bbd3dd68437e6ff131583719a032d48"}
Apr 16 16:08:43.488817 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:43.488820    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db74f6b9d-2qz9p" event={"ID":"5461e083-38c9-41c5-9e80-3fb2f3603562","Type":"ContainerStarted","Data":"504435ed88a66bab8e2ac41db182ad0a78adcf073cdff8aeefa1ffb449ad1e81"}
Apr 16 16:08:43.520041 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:43.519936    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5db74f6b9d-2qz9p" podStartSLOduration=1.5199110120000001 podStartE2EDuration="1.519911012s" podCreationTimestamp="2026-04-16 16:08:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:08:43.512513406 +0000 UTC m=+444.908597823" watchObservedRunningTime="2026-04-16 16:08:43.519911012 +0000 UTC m=+444.915995434"
Apr 16 16:08:52.876995 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:52.876917    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:52.876995 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:52.877004    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:52.882271 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:52.882236    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:53.526850 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:53.526802    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db74f6b9d-2qz9p"
Apr 16 16:08:53.577586 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:08:53.577530    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4dccb4d-gsqx4"]
Apr 16 16:09:18.601751 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.601648    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-56b4dccb4d-gsqx4" podUID="9cd41e70-18c1-4069-a0a1-c4c318bc942a" containerName="console" containerID="cri-o://1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7" gracePeriod=15
Apr 16 16:09:18.869113 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.869078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56b4dccb4d-gsqx4_9cd41e70-18c1-4069-a0a1-c4c318bc942a/console/0.log"
Apr 16 16:09:18.869347 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.869190    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:09:18.954659 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.954597    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q62mc\" (UniqueName: \"kubernetes.io/projected/9cd41e70-18c1-4069-a0a1-c4c318bc942a-kube-api-access-q62mc\") pod \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") "
Apr 16 16:09:18.954659 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.954664    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-oauth-serving-cert\") pod \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") "
Apr 16 16:09:18.955007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.954686    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-serving-cert\") pod \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") "
Apr 16 16:09:18.955007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.954812    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-trusted-ca-bundle\") pod \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") "
Apr 16 16:09:18.955007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.954858    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-config\") pod \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") "
Apr 16 16:09:18.955007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.954889    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-oauth-config\") pod \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") "
Apr 16 16:09:18.955007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.954926    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-service-ca\") pod \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\" (UID: \"9cd41e70-18c1-4069-a0a1-c4c318bc942a\") "
Apr 16 16:09:18.955269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.955227    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9cd41e70-18c1-4069-a0a1-c4c318bc942a" (UID: "9cd41e70-18c1-4069-a0a1-c4c318bc942a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:09:18.955376 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.955293    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9cd41e70-18c1-4069-a0a1-c4c318bc942a" (UID: "9cd41e70-18c1-4069-a0a1-c4c318bc942a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:09:18.955376 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.955321    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-config" (OuterVolumeSpecName: "console-config") pod "9cd41e70-18c1-4069-a0a1-c4c318bc942a" (UID: "9cd41e70-18c1-4069-a0a1-c4c318bc942a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:09:18.955482 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.955401    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-service-ca" (OuterVolumeSpecName: "service-ca") pod "9cd41e70-18c1-4069-a0a1-c4c318bc942a" (UID: "9cd41e70-18c1-4069-a0a1-c4c318bc942a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:09:18.957250 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.957220    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9cd41e70-18c1-4069-a0a1-c4c318bc942a" (UID: "9cd41e70-18c1-4069-a0a1-c4c318bc942a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:09:18.957361 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.957333    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd41e70-18c1-4069-a0a1-c4c318bc942a-kube-api-access-q62mc" (OuterVolumeSpecName: "kube-api-access-q62mc") pod "9cd41e70-18c1-4069-a0a1-c4c318bc942a" (UID: "9cd41e70-18c1-4069-a0a1-c4c318bc942a"). InnerVolumeSpecName "kube-api-access-q62mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:09:18.957361 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:18.957346    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9cd41e70-18c1-4069-a0a1-c4c318bc942a" (UID: "9cd41e70-18c1-4069-a0a1-c4c318bc942a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:09:19.055840 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.055800    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62mc\" (UniqueName: \"kubernetes.io/projected/9cd41e70-18c1-4069-a0a1-c4c318bc942a-kube-api-access-q62mc\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:09:19.055840 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.055830    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-oauth-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:09:19.055840 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.055841    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-serving-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:09:19.055840 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.055851    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-trusted-ca-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:09:19.056143 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.055861    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:09:19.056143 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.055870    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cd41e70-18c1-4069-a0a1-c4c318bc942a-console-oauth-config\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:09:19.056143 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.055879    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd41e70-18c1-4069-a0a1-c4c318bc942a-service-ca\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:09:19.620355 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.620298    2578 generic.go:358] "Generic (PLEG): container finished" podID="9cd41e70-18c1-4069-a0a1-c4c318bc942a" containerID="1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7" exitCode=2
Apr 16 16:09:19.621019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.620393    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4dccb4d-gsqx4" event={"ID":"9cd41e70-18c1-4069-a0a1-c4c318bc942a","Type":"ContainerDied","Data":"1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7"}
Apr 16 16:09:19.621019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.620424    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b4dccb4d-gsqx4"
Apr 16 16:09:19.621019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.620493    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b4dccb4d-gsqx4" event={"ID":"9cd41e70-18c1-4069-a0a1-c4c318bc942a","Type":"ContainerDied","Data":"0a1abaad544de4f6a59360c83c651c4527f9e84383fbce2c269065d81a493335"}
Apr 16 16:09:19.621019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.620520    2578 scope.go:117] "RemoveContainer" containerID="1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7"
Apr 16 16:09:19.632799 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.632744    2578 scope.go:117] "RemoveContainer" containerID="1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7"
Apr 16 16:09:19.633186 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:09:19.633161    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7\": container with ID starting with 1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7 not found: ID does not exist" containerID="1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7"
Apr 16 16:09:19.633294 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.633200    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7"} err="failed to get container status \"1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7\": rpc error: code = NotFound desc = could not find container \"1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7\": container with ID starting with 1f9a41f54eb921687edc372bd874b6a7ae5748cdb06c080bd0d441d0f3f2a6a7 not found: ID does not exist"
Apr 16 16:09:19.655612 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.655557    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b4dccb4d-gsqx4"]
Apr 16 16:09:19.659221 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:19.659174    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56b4dccb4d-gsqx4"]
Apr 16 16:09:21.107012 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:09:21.106971    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd41e70-18c1-4069-a0a1-c4c318bc942a" path="/var/lib/kubelet/pods/9cd41e70-18c1-4069-a0a1-c4c318bc942a/volumes"
Apr 16 16:11:19.042921 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:11:19.042888    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:11:19.042921 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:11:19.042897    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:13:41.910251 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.910208    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"]
Apr 16 16:13:41.912673 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.910569    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd41e70-18c1-4069-a0a1-c4c318bc942a" containerName="console"
Apr 16 16:13:41.912673 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.910582    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd41e70-18c1-4069-a0a1-c4c318bc942a" containerName="console"
Apr 16 16:13:41.912673 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.910665    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="9cd41e70-18c1-4069-a0a1-c4c318bc942a" containerName="console"
Apr 16 16:13:41.913613 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.913595    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:41.916264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.916241    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 16 16:13:41.917087 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.917071    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 16 16:13:41.917148 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.917075    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9wlr7\""
Apr 16 16:13:41.929805 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.929772    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"]
Apr 16 16:13:41.942644 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.942612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5l5m\" (UniqueName: \"kubernetes.io/projected/1ff04cdb-8a32-4986-ae83-040f89191783-kube-api-access-z5l5m\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:41.942830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.942695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:41.942830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:41.942726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.043146 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.043107    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z5l5m\" (UniqueName: \"kubernetes.io/projected/1ff04cdb-8a32-4986-ae83-040f89191783-kube-api-access-z5l5m\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.043347 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.043190    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.043347 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.043214    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.043582 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.043568    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.043636 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.043615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.051868 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.051846    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5l5m\" (UniqueName: \"kubernetes.io/projected/1ff04cdb-8a32-4986-ae83-040f89191783-kube-api-access-z5l5m\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.223002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.222968    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:13:42.349920 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.349896    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"]
Apr 16 16:13:42.352643 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:13:42.352601    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff04cdb_8a32_4986_ae83_040f89191783.slice/crio-209d1b5da2d1aef80dbd19a91c91c461338fc99c766ff95ad3b07342414bd184 WatchSource:0}: Error finding container 209d1b5da2d1aef80dbd19a91c91c461338fc99c766ff95ad3b07342414bd184: Status 404 returned error can't find the container with id 209d1b5da2d1aef80dbd19a91c91c461338fc99c766ff95ad3b07342414bd184
Apr 16 16:13:42.442889 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:42.442850    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t" event={"ID":"1ff04cdb-8a32-4986-ae83-040f89191783","Type":"ContainerStarted","Data":"209d1b5da2d1aef80dbd19a91c91c461338fc99c766ff95ad3b07342414bd184"}
Apr 16 16:13:50.468559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:50.468521    2578 generic.go:358] "Generic (PLEG): container finished" podID="1ff04cdb-8a32-4986-ae83-040f89191783" containerID="7616fd3bca1b0a8eb3e5e090bc117b0a9d793179c71c1cb5a17ad41612af4ddd" exitCode=0
Apr 16 16:13:50.468900 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:50.468600    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t" event={"ID":"1ff04cdb-8a32-4986-ae83-040f89191783","Type":"ContainerDied","Data":"7616fd3bca1b0a8eb3e5e090bc117b0a9d793179c71c1cb5a17ad41612af4ddd"}
Apr 16 16:13:50.469596 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:50.469581    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:13:52.476681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:52.476646    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t" event={"ID":"1ff04cdb-8a32-4986-ae83-040f89191783","Type":"ContainerStarted","Data":"753a51838680e1897dc03d07e844b16c6716c323a33b6dda18b48f000ed46f49"}
Apr 16 16:13:53.481299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:53.481266    2578 generic.go:358] "Generic (PLEG): container finished" podID="1ff04cdb-8a32-4986-ae83-040f89191783" containerID="753a51838680e1897dc03d07e844b16c6716c323a33b6dda18b48f000ed46f49" exitCode=0
Apr 16 16:13:53.481807 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:53.481339    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t" event={"ID":"1ff04cdb-8a32-4986-ae83-040f89191783","Type":"ContainerDied","Data":"753a51838680e1897dc03d07e844b16c6716c323a33b6dda18b48f000ed46f49"}
Apr 16 16:13:59.500241 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:59.500208    2578 generic.go:358] "Generic (PLEG): container finished" podID="1ff04cdb-8a32-4986-ae83-040f89191783" containerID="131d59e0f946ed32533ad454b9f401191e8484ec781402e07d956ae404d3c2c8" exitCode=0
Apr 16 16:13:59.500608 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:13:59.500293    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t" event={"ID":"1ff04cdb-8a32-4986-ae83-040f89191783","Type":"ContainerDied","Data":"131d59e0f946ed32533ad454b9f401191e8484ec781402e07d956ae404d3c2c8"}
Apr 16 16:14:00.625726 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.625693    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:14:00.711292 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.711260    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5l5m\" (UniqueName: \"kubernetes.io/projected/1ff04cdb-8a32-4986-ae83-040f89191783-kube-api-access-z5l5m\") pod \"1ff04cdb-8a32-4986-ae83-040f89191783\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") "
Apr 16 16:14:00.711292 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.711294    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-util\") pod \"1ff04cdb-8a32-4986-ae83-040f89191783\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") "
Apr 16 16:14:00.711565 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.711334    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-bundle\") pod \"1ff04cdb-8a32-4986-ae83-040f89191783\" (UID: \"1ff04cdb-8a32-4986-ae83-040f89191783\") "
Apr 16 16:14:00.711927 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.711903    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-bundle" (OuterVolumeSpecName: "bundle") pod "1ff04cdb-8a32-4986-ae83-040f89191783" (UID: "1ff04cdb-8a32-4986-ae83-040f89191783"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:14:00.713750 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.713726    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff04cdb-8a32-4986-ae83-040f89191783-kube-api-access-z5l5m" (OuterVolumeSpecName: "kube-api-access-z5l5m") pod "1ff04cdb-8a32-4986-ae83-040f89191783" (UID: "1ff04cdb-8a32-4986-ae83-040f89191783"). InnerVolumeSpecName "kube-api-access-z5l5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:14:00.715599 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.715572    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-util" (OuterVolumeSpecName: "util") pod "1ff04cdb-8a32-4986-ae83-040f89191783" (UID: "1ff04cdb-8a32-4986-ae83-040f89191783"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:14:00.812066 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.811984    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5l5m\" (UniqueName: \"kubernetes.io/projected/1ff04cdb-8a32-4986-ae83-040f89191783-kube-api-access-z5l5m\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:14:00.812066 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.812011    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:14:00.812066 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:00.812021    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ff04cdb-8a32-4986-ae83-040f89191783-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:14:01.506880 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:01.506849    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t" event={"ID":"1ff04cdb-8a32-4986-ae83-040f89191783","Type":"ContainerDied","Data":"209d1b5da2d1aef80dbd19a91c91c461338fc99c766ff95ad3b07342414bd184"}
Apr 16 16:14:01.506880 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:01.506879    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="209d1b5da2d1aef80dbd19a91c91c461338fc99c766ff95ad3b07342414bd184"
Apr 16 16:14:01.507081 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:01.506896    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cg5n6t"
Apr 16 16:14:03.530050 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530017    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"]
Apr 16 16:14:03.530435 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530381    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff04cdb-8a32-4986-ae83-040f89191783" containerName="util"
Apr 16 16:14:03.530435 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530398    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff04cdb-8a32-4986-ae83-040f89191783" containerName="util"
Apr 16 16:14:03.530435 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530409    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff04cdb-8a32-4986-ae83-040f89191783" containerName="pull"
Apr 16 16:14:03.530435 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530419    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff04cdb-8a32-4986-ae83-040f89191783" containerName="pull"
Apr 16 16:14:03.530614 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530464    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff04cdb-8a32-4986-ae83-040f89191783" containerName="extract"
Apr 16 16:14:03.530614 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530473    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff04cdb-8a32-4986-ae83-040f89191783" containerName="extract"
Apr 16 16:14:03.530614 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.530533    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ff04cdb-8a32-4986-ae83-040f89191783" containerName="extract"
Apr 16 16:14:03.533833 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.533808    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.536104 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.536085    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Apr 16 16:14:03.536228 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.536117    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Apr 16 16:14:03.536228 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.536116    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-nkhpv\""
Apr 16 16:14:03.536326 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.536249    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Apr 16 16:14:03.542710 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.542681    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"]
Apr 16 16:14:03.635128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.635097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cs9v\" (UniqueName: \"kubernetes.io/projected/f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf-kube-api-access-5cs9v\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5\" (UID: \"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.635301 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.635146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5\" (UID: \"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.736417 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.736375    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5\" (UID: \"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.736586 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.736490    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5cs9v\" (UniqueName: \"kubernetes.io/projected/f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf-kube-api-access-5cs9v\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5\" (UID: \"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.739560 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.739538    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5\" (UID: \"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.744748 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.744724    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cs9v\" (UniqueName: \"kubernetes.io/projected/f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf-kube-api-access-5cs9v\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5\" (UID: \"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.844657 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.844573    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:03.979286 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:03.979241    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"]
Apr 16 16:14:03.982997 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:14:03.982967    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60384de_b2f2_4fa2_9b4b_2e28dc6c1daf.slice/crio-92f227ad29888f185e9a5ba6541536b07ba826749e0a99e0b525eb71a4768061 WatchSource:0}: Error finding container 92f227ad29888f185e9a5ba6541536b07ba826749e0a99e0b525eb71a4768061: Status 404 returned error can't find the container with id 92f227ad29888f185e9a5ba6541536b07ba826749e0a99e0b525eb71a4768061
Apr 16 16:14:04.517409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:04.517373    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5" event={"ID":"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf","Type":"ContainerStarted","Data":"92f227ad29888f185e9a5ba6541536b07ba826749e0a99e0b525eb71a4768061"}
Apr 16 16:14:10.452245 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.452215    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-wckzl"]
Apr 16 16:14:10.455519 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.455504    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.459088 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.459063    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Apr 16 16:14:10.459234 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.459067    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Apr 16 16:14:10.459299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.459076    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tqzkq\""
Apr 16 16:14:10.466372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.466352    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-wckzl"]
Apr 16 16:14:10.543392 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.543361    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5" event={"ID":"f60384de-b2f2-4fa2-9b4b-2e28dc6c1daf","Type":"ContainerStarted","Data":"59c2a6c72f0c02d9c9057427c6bed3dc57a9f37f8b8a44bec4ab02d4a2524e80"}
Apr 16 16:14:10.543524 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.543424    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:10.562379 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.562336    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5" podStartSLOduration=1.6090754409999999 podStartE2EDuration="7.562324812s" podCreationTimestamp="2026-04-16 16:14:03 +0000 UTC" firstStartedPulling="2026-04-16 16:14:03.985385375 +0000 UTC m=+765.381469773" lastFinishedPulling="2026-04-16 16:14:09.938634745 +0000 UTC m=+771.334719144" observedRunningTime="2026-04-16 16:14:10.560349873 +0000 UTC m=+771.956434293" watchObservedRunningTime="2026-04-16 16:14:10.562324812 +0000 UTC m=+771.958409232"
Apr 16 16:14:10.593074 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.593047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.593154 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.593087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m98\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-kube-api-access-n9m98\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.593192 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.593168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/372de5aa-8541-4dc2-83b8-3499dd114775-cabundle0\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.694387 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.694341    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m98\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-kube-api-access-n9m98\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.694524 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.694437    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/372de5aa-8541-4dc2-83b8-3499dd114775-cabundle0\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.694524 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.694516    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.694640 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.694621    2578 secret.go:281] references non-existent secret key: ca.crt
Apr 16 16:14:10.694640 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.694636    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Apr 16 16:14:10.694719 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.694646    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-wckzl: references non-existent secret key: ca.crt
Apr 16 16:14:10.694719 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.694707    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates podName:372de5aa-8541-4dc2-83b8-3499dd114775 nodeName:}" failed. No retries permitted until 2026-04-16 16:14:11.194687991 +0000 UTC m=+772.590772405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates") pod "keda-operator-ffbb595cb-wckzl" (UID: "372de5aa-8541-4dc2-83b8-3499dd114775") : references non-existent secret key: ca.crt
Apr 16 16:14:10.695311 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.695282    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/372de5aa-8541-4dc2-83b8-3499dd114775-cabundle0\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.703416 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.703361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m98\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-kube-api-access-n9m98\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:10.763653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.763629    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"]
Apr 16 16:14:10.766890 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.766875    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:10.769029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.769008    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Apr 16 16:14:10.774501 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.774471    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"]
Apr 16 16:14:10.895390 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.895358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46l5\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-kube-api-access-s46l5\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:10.895538 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.895403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:10.895538 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.895428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:10.984407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.984370    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-tcpmv"]
Apr 16 16:14:10.987532 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.987517    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:10.989952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.989931    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Apr 16 16:14:10.994974 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.994953    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-tcpmv"]
Apr 16 16:14:10.996185 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.996167    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s46l5\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-kube-api-access-s46l5\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:10.996262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.996197    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:10.996262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.996229    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:10.996333 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.996315    2578 secret.go:281] references non-existent secret key: tls.crt
Apr 16 16:14:10.996373 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.996332    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 16 16:14:10.996373 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.996352    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g: references non-existent secret key: tls.crt
Apr 16 16:14:10.996462 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:10.996412    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates podName:b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba nodeName:}" failed. No retries permitted until 2026-04-16 16:14:11.496394283 +0000 UTC m=+772.892478695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates") pod "keda-metrics-apiserver-7c9f485588-5s72g" (UID: "b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba") : references non-existent secret key: tls.crt
Apr 16 16:14:10.996667 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:10.996651    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:11.004800 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.004779    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46l5\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-kube-api-access-s46l5\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:11.097366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.097341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/06bf9952-1342-4619-bcad-830def194155-certificates\") pod \"keda-admission-cf49989db-tcpmv\" (UID: \"06bf9952-1342-4619-bcad-830def194155\") " pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:11.097541 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.097373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvmbc\" (UniqueName: \"kubernetes.io/projected/06bf9952-1342-4619-bcad-830def194155-kube-api-access-fvmbc\") pod \"keda-admission-cf49989db-tcpmv\" (UID: \"06bf9952-1342-4619-bcad-830def194155\") " pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:11.198249 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.198211    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:11.198423 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.198267    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/06bf9952-1342-4619-bcad-830def194155-certificates\") pod \"keda-admission-cf49989db-tcpmv\" (UID: \"06bf9952-1342-4619-bcad-830def194155\") " pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:11.198423 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.198299    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fvmbc\" (UniqueName: \"kubernetes.io/projected/06bf9952-1342-4619-bcad-830def194155-kube-api-access-fvmbc\") pod \"keda-admission-cf49989db-tcpmv\" (UID: \"06bf9952-1342-4619-bcad-830def194155\") " pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:11.198423 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.198355    2578 secret.go:281] references non-existent secret key: ca.crt
Apr 16 16:14:11.198423 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.198371    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Apr 16 16:14:11.198423 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.198381    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-wckzl: references non-existent secret key: ca.crt
Apr 16 16:14:11.198714 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.198441    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates podName:372de5aa-8541-4dc2-83b8-3499dd114775 nodeName:}" failed. No retries permitted until 2026-04-16 16:14:12.198426865 +0000 UTC m=+773.594511263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates") pod "keda-operator-ffbb595cb-wckzl" (UID: "372de5aa-8541-4dc2-83b8-3499dd114775") : references non-existent secret key: ca.crt
Apr 16 16:14:11.201277 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.201246    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/06bf9952-1342-4619-bcad-830def194155-certificates\") pod \"keda-admission-cf49989db-tcpmv\" (UID: \"06bf9952-1342-4619-bcad-830def194155\") " pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:11.208416 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.208368    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvmbc\" (UniqueName: \"kubernetes.io/projected/06bf9952-1342-4619-bcad-830def194155-kube-api-access-fvmbc\") pod \"keda-admission-cf49989db-tcpmv\" (UID: \"06bf9952-1342-4619-bcad-830def194155\") " pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:11.299014 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.298927    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:11.445581 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.445469    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-tcpmv"]
Apr 16 16:14:11.448187 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:14:11.448155    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bf9952_1342_4619_bcad_830def194155.slice/crio-43446b5e7bdaa7be12a77af2f4a3a88b6bed008362fa5844bb24fed78d90f409 WatchSource:0}: Error finding container 43446b5e7bdaa7be12a77af2f4a3a88b6bed008362fa5844bb24fed78d90f409: Status 404 returned error can't find the container with id 43446b5e7bdaa7be12a77af2f4a3a88b6bed008362fa5844bb24fed78d90f409
Apr 16 16:14:11.501333 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.501303    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:11.501746 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.501425    2578 secret.go:281] references non-existent secret key: tls.crt
Apr 16 16:14:11.501746 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.501464    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 16 16:14:11.501746 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.501494    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g: references non-existent secret key: tls.crt
Apr 16 16:14:11.501746 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:11.501556    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates podName:b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba nodeName:}" failed. No retries permitted until 2026-04-16 16:14:12.501537689 +0000 UTC m=+773.897622091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates") pod "keda-metrics-apiserver-7c9f485588-5s72g" (UID: "b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba") : references non-existent secret key: tls.crt
Apr 16 16:14:11.547497 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:11.547472    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-tcpmv" event={"ID":"06bf9952-1342-4619-bcad-830def194155","Type":"ContainerStarted","Data":"43446b5e7bdaa7be12a77af2f4a3a88b6bed008362fa5844bb24fed78d90f409"}
Apr 16 16:14:12.208093 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:12.208052    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:12.208291 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.208200    2578 secret.go:281] references non-existent secret key: ca.crt
Apr 16 16:14:12.208291 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.208218    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Apr 16 16:14:12.208291 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.208227    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-wckzl: references non-existent secret key: ca.crt
Apr 16 16:14:12.208291 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.208282    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates podName:372de5aa-8541-4dc2-83b8-3499dd114775 nodeName:}" failed. No retries permitted until 2026-04-16 16:14:14.208268373 +0000 UTC m=+775.604352776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates") pod "keda-operator-ffbb595cb-wckzl" (UID: "372de5aa-8541-4dc2-83b8-3499dd114775") : references non-existent secret key: ca.crt
Apr 16 16:14:12.510717 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:12.510676    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:12.511213 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.510841    2578 secret.go:281] references non-existent secret key: tls.crt
Apr 16 16:14:12.511213 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.510868    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Apr 16 16:14:12.511213 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.510890    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g: references non-existent secret key: tls.crt
Apr 16 16:14:12.511213 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:14:12.510949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates podName:b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba nodeName:}" failed. No retries permitted until 2026-04-16 16:14:14.510930566 +0000 UTC m=+775.907014978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates") pod "keda-metrics-apiserver-7c9f485588-5s72g" (UID: "b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba") : references non-existent secret key: tls.crt
Apr 16 16:14:13.554534 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:13.554500    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-tcpmv" event={"ID":"06bf9952-1342-4619-bcad-830def194155","Type":"ContainerStarted","Data":"427be9ebf0ce45d8f6f7bb35c9ec92a7a4af58a237d53916dd4756aa9cf56f30"}
Apr 16 16:14:13.554876 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:13.554552    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:13.572104 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:13.572017    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-tcpmv" podStartSLOduration=1.7062239 podStartE2EDuration="3.572003435s" podCreationTimestamp="2026-04-16 16:14:10 +0000 UTC" firstStartedPulling="2026-04-16 16:14:11.449542853 +0000 UTC m=+772.845627270" lastFinishedPulling="2026-04-16 16:14:13.315322394 +0000 UTC m=+774.711406805" observedRunningTime="2026-04-16 16:14:13.570788687 +0000 UTC m=+774.966873108" watchObservedRunningTime="2026-04-16 16:14:13.572003435 +0000 UTC m=+774.968087854"
Apr 16 16:14:14.226601 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.226571    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:14.229189 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.229156    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/372de5aa-8541-4dc2-83b8-3499dd114775-certificates\") pod \"keda-operator-ffbb595cb-wckzl\" (UID: \"372de5aa-8541-4dc2-83b8-3499dd114775\") " pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:14.366405 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.366380    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:14.495663 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.495641    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-wckzl"]
Apr 16 16:14:14.497369 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:14:14.497339    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372de5aa_8541_4dc2_83b8_3499dd114775.slice/crio-3affc5fb026fd0f8502a76db9fd0f939ccd9139c607b63cccb8fdc053ff43fa8 WatchSource:0}: Error finding container 3affc5fb026fd0f8502a76db9fd0f939ccd9139c607b63cccb8fdc053ff43fa8: Status 404 returned error can't find the container with id 3affc5fb026fd0f8502a76db9fd0f939ccd9139c607b63cccb8fdc053ff43fa8
Apr 16 16:14:14.529508 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.529478    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:14.532177 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.532149    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s72g\" (UID: \"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:14.558338 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.558307    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-wckzl" event={"ID":"372de5aa-8541-4dc2-83b8-3499dd114775","Type":"ContainerStarted","Data":"3affc5fb026fd0f8502a76db9fd0f939ccd9139c607b63cccb8fdc053ff43fa8"}
Apr 16 16:14:14.677783 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.677758    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:14.793139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:14.793112    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"]
Apr 16 16:14:14.795439 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:14:14.795399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08ea9f5_c7ce_4da6_9a4e_8d0f89356aba.slice/crio-06b3f146225cec8a43b0eb7dd9fe82fbd27c7db4ad6875f4d50bcaf0b9cd11b8 WatchSource:0}: Error finding container 06b3f146225cec8a43b0eb7dd9fe82fbd27c7db4ad6875f4d50bcaf0b9cd11b8: Status 404 returned error can't find the container with id 06b3f146225cec8a43b0eb7dd9fe82fbd27c7db4ad6875f4d50bcaf0b9cd11b8
Apr 16 16:14:15.563340 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:15.563301    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g" event={"ID":"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba","Type":"ContainerStarted","Data":"06b3f146225cec8a43b0eb7dd9fe82fbd27c7db4ad6875f4d50bcaf0b9cd11b8"}
Apr 16 16:14:20.582680 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:20.582637    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g" event={"ID":"b08ea9f5-c7ce-4da6-9a4e-8d0f89356aba","Type":"ContainerStarted","Data":"c72e489fd3ea44fa5c96818f4f85dbfb2c16cbe1958f779bf704256679c8b3eb"}
Apr 16 16:14:20.583120 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:20.582803    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:20.584058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:20.584025    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-wckzl" event={"ID":"372de5aa-8541-4dc2-83b8-3499dd114775","Type":"ContainerStarted","Data":"26c6ba06c80c9e71aeec030d9d5eb5c340c29adf823fd255bd1e8ff52a4b895a"}
Apr 16 16:14:20.584177 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:20.584077    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:14:20.598281 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:20.598246    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g" podStartSLOduration=5.818471928 podStartE2EDuration="10.598236101s" podCreationTimestamp="2026-04-16 16:14:10 +0000 UTC" firstStartedPulling="2026-04-16 16:14:14.796766554 +0000 UTC m=+776.192850967" lastFinishedPulling="2026-04-16 16:14:19.576530742 +0000 UTC m=+780.972615140" observedRunningTime="2026-04-16 16:14:20.597005019 +0000 UTC m=+781.993089439" watchObservedRunningTime="2026-04-16 16:14:20.598236101 +0000 UTC m=+781.994320520"
Apr 16 16:14:20.610823 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:20.610780    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-wckzl" podStartSLOduration=5.5283050209999995 podStartE2EDuration="10.610770787s" podCreationTimestamp="2026-04-16 16:14:10 +0000 UTC" firstStartedPulling="2026-04-16 16:14:14.498775112 +0000 UTC m=+775.894859524" lastFinishedPulling="2026-04-16 16:14:19.581240876 +0000 UTC m=+780.977325290" observedRunningTime="2026-04-16 16:14:20.610603426 +0000 UTC m=+782.006687846" watchObservedRunningTime="2026-04-16 16:14:20.610770787 +0000 UTC m=+782.006855236"
Apr 16 16:14:31.549639 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:31.549569    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-t8gz5"
Apr 16 16:14:31.592258 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:31.592229    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s72g"
Apr 16 16:14:34.561533 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:34.561496    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-tcpmv"
Apr 16 16:14:41.589870 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:14:41.589832    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-wckzl"
Apr 16 16:15:05.186933 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.186883    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"]
Apr 16 16:15:05.190683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.190661    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.193269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.193240    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 16 16:15:05.193269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.193240    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9wlr7\""
Apr 16 16:15:05.194239 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.194211    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 16 16:15:05.197016 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.196985    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"]
Apr 16 16:15:05.268583 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.268525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-bundle\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.268899 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.268602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnvr\" (UniqueName: \"kubernetes.io/projected/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-kube-api-access-6hnvr\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.268899 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.268751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-util\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.369987 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.369938    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-bundle\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.370219 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.370015    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnvr\" (UniqueName: \"kubernetes.io/projected/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-kube-api-access-6hnvr\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.370219 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.370072    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-util\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.370411 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.370389    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-bundle\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.370473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.370418    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-util\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.378644 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.378616    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnvr\" (UniqueName: \"kubernetes.io/projected/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-kube-api-access-6hnvr\") pod \"925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") " pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.509619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.509557    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:05.654526 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.654284    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"]
Apr 16 16:15:05.658040 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:15:05.658001    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd100f82f_f176_40f9_a1bb_9c6c1d454a1b.slice/crio-d98b79621175e8ca34876a87b57ca92159b3a6cfaed9a6b73ee94cf62f463da9 WatchSource:0}: Error finding container d98b79621175e8ca34876a87b57ca92159b3a6cfaed9a6b73ee94cf62f463da9: Status 404 returned error can't find the container with id d98b79621175e8ca34876a87b57ca92159b3a6cfaed9a6b73ee94cf62f463da9
Apr 16 16:15:05.752214 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.752149    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww" event={"ID":"d100f82f-f176-40f9-a1bb-9c6c1d454a1b","Type":"ContainerStarted","Data":"05027037558d396880d280c4ec691605a57d8a9fbc5123857f67e296e0c1125e"}
Apr 16 16:15:05.752214 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:05.752216    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww" event={"ID":"d100f82f-f176-40f9-a1bb-9c6c1d454a1b","Type":"ContainerStarted","Data":"d98b79621175e8ca34876a87b57ca92159b3a6cfaed9a6b73ee94cf62f463da9"}
Apr 16 16:15:06.758020 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:06.757968    2578 generic.go:358] "Generic (PLEG): container finished" podID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerID="05027037558d396880d280c4ec691605a57d8a9fbc5123857f67e296e0c1125e" exitCode=0
Apr 16 16:15:06.758698 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:06.758068    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww" event={"ID":"d100f82f-f176-40f9-a1bb-9c6c1d454a1b","Type":"ContainerDied","Data":"05027037558d396880d280c4ec691605a57d8a9fbc5123857f67e296e0c1125e"}
Apr 16 16:15:07.764178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:07.764089    2578 generic.go:358] "Generic (PLEG): container finished" podID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerID="25fc243b17ee404fde6970635c6eafdb2482de937c46db866cf1572087c3a111" exitCode=0
Apr 16 16:15:07.764545 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:07.764183    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww" event={"ID":"d100f82f-f176-40f9-a1bb-9c6c1d454a1b","Type":"ContainerDied","Data":"25fc243b17ee404fde6970635c6eafdb2482de937c46db866cf1572087c3a111"}
Apr 16 16:15:08.770359 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:08.770317    2578 generic.go:358] "Generic (PLEG): container finished" podID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerID="ccee19c75b50e6db8cb44d81c45a59eaa2fd4efad58c8ef68b3a1b03f786ce82" exitCode=0
Apr 16 16:15:08.770823 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:08.770391    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww" event={"ID":"d100f82f-f176-40f9-a1bb-9c6c1d454a1b","Type":"ContainerDied","Data":"ccee19c75b50e6db8cb44d81c45a59eaa2fd4efad58c8ef68b3a1b03f786ce82"}
Apr 16 16:15:09.922580 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:09.922548    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:10.019019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.018974    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-util\") pod \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") "
Apr 16 16:15:10.019299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.019035    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hnvr\" (UniqueName: \"kubernetes.io/projected/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-kube-api-access-6hnvr\") pod \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") "
Apr 16 16:15:10.019299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.019152    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-bundle\") pod \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\" (UID: \"d100f82f-f176-40f9-a1bb-9c6c1d454a1b\") "
Apr 16 16:15:10.019979 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.019945    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-bundle" (OuterVolumeSpecName: "bundle") pod "d100f82f-f176-40f9-a1bb-9c6c1d454a1b" (UID: "d100f82f-f176-40f9-a1bb-9c6c1d454a1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:15:10.021741 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.021711    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-kube-api-access-6hnvr" (OuterVolumeSpecName: "kube-api-access-6hnvr") pod "d100f82f-f176-40f9-a1bb-9c6c1d454a1b" (UID: "d100f82f-f176-40f9-a1bb-9c6c1d454a1b"). InnerVolumeSpecName "kube-api-access-6hnvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:15:10.025265 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.025164    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-util" (OuterVolumeSpecName: "util") pod "d100f82f-f176-40f9-a1bb-9c6c1d454a1b" (UID: "d100f82f-f176-40f9-a1bb-9c6c1d454a1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:15:10.120859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.120806    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:10.120859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.120849    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hnvr\" (UniqueName: \"kubernetes.io/projected/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-kube-api-access-6hnvr\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:10.120859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.120867    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d100f82f-f176-40f9-a1bb-9c6c1d454a1b-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:10.781051 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.780998    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww" event={"ID":"d100f82f-f176-40f9-a1bb-9c6c1d454a1b","Type":"ContainerDied","Data":"d98b79621175e8ca34876a87b57ca92159b3a6cfaed9a6b73ee94cf62f463da9"}
Apr 16 16:15:10.781051 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.781048    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d98b79621175e8ca34876a87b57ca92159b3a6cfaed9a6b73ee94cf62f463da9"
Apr 16 16:15:10.781384 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:10.781118    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5mrkww"
Apr 16 16:15:17.953754 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.953684    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"]
Apr 16 16:15:17.954616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.954355    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerName="pull"
Apr 16 16:15:17.954616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.954385    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerName="pull"
Apr 16 16:15:17.954616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.954403    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerName="extract"
Apr 16 16:15:17.954616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.954414    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerName="extract"
Apr 16 16:15:17.954616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.954477    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerName="util"
Apr 16 16:15:17.954616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.954487    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerName="util"
Apr 16 16:15:17.954616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.954602    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="d100f82f-f176-40f9-a1bb-9c6c1d454a1b" containerName="extract"
Apr 16 16:15:17.959601 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.959562    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:17.962291 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.962259    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Apr 16 16:15:17.962291 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.962255    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Apr 16 16:15:17.962559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.962255    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-drstw\""
Apr 16 16:15:17.968678 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.968637    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"]
Apr 16 16:15:17.998654 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.998603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded-tmp\") pod \"cert-manager-operator-controller-manager-7ccfb878b5-xgxm9\" (UID: \"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:17.998924 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:17.998757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded-kube-api-access-vprft\") pod \"cert-manager-operator-controller-manager-7ccfb878b5-xgxm9\" (UID: \"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:18.099968 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:18.099885    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded-kube-api-access-vprft\") pod \"cert-manager-operator-controller-manager-7ccfb878b5-xgxm9\" (UID: \"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:18.100322 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:18.100029    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded-tmp\") pod \"cert-manager-operator-controller-manager-7ccfb878b5-xgxm9\" (UID: \"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:18.100644 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:18.100623    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded-tmp\") pod \"cert-manager-operator-controller-manager-7ccfb878b5-xgxm9\" (UID: \"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:18.109687 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:18.109643    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprft\" (UniqueName: \"kubernetes.io/projected/e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded-kube-api-access-vprft\") pod \"cert-manager-operator-controller-manager-7ccfb878b5-xgxm9\" (UID: \"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:18.273617 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:18.273552    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"
Apr 16 16:15:18.439877 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:18.439731    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9"]
Apr 16 16:15:18.444433 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:15:18.444385    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f6b1be_1161_4b31_a8cd_da2e6e2e7ded.slice/crio-170dae2278c49dd29bb1f0e3d3ffeee3aa8e88e405e1d2ff423b3e50583aa8bc WatchSource:0}: Error finding container 170dae2278c49dd29bb1f0e3d3ffeee3aa8e88e405e1d2ff423b3e50583aa8bc: Status 404 returned error can't find the container with id 170dae2278c49dd29bb1f0e3d3ffeee3aa8e88e405e1d2ff423b3e50583aa8bc
Apr 16 16:15:18.814931 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:18.814884    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9" event={"ID":"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded","Type":"ContainerStarted","Data":"170dae2278c49dd29bb1f0e3d3ffeee3aa8e88e405e1d2ff423b3e50583aa8bc"}
Apr 16 16:15:20.826472 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:20.826396    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9" event={"ID":"e7f6b1be-1161-4b31-a8cd-da2e6e2e7ded","Type":"ContainerStarted","Data":"0ff3f471893a43a0b9826ca7b82bf2311403c247c0d781cec21a1c059fedc16c"}
Apr 16 16:15:20.846379 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:20.846291    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7ccfb878b5-xgxm9" podStartSLOduration=1.920248908 podStartE2EDuration="3.846263573s" podCreationTimestamp="2026-04-16 16:15:17 +0000 UTC" firstStartedPulling="2026-04-16 16:15:18.448392988 +0000 UTC m=+839.844477385" lastFinishedPulling="2026-04-16 16:15:20.374407649 +0000 UTC m=+841.770492050" observedRunningTime="2026-04-16 16:15:20.844909518 +0000 UTC m=+842.240993929" watchObservedRunningTime="2026-04-16 16:15:20.846263573 +0000 UTC m=+842.242347992"
Apr 16 16:15:25.278026 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.277955    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"]
Apr 16 16:15:25.282800 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.282773    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.285922 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.285830    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9wlr7\""
Apr 16 16:15:25.286138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.285926    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 16 16:15:25.286138 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.285830    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 16 16:15:25.289210 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.289173    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"]
Apr 16 16:15:25.366315 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.366248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlckn\" (UniqueName: \"kubernetes.io/projected/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-kube-api-access-wlckn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.366630 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.366410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.366630 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.366531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.467990 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.467928    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.468267 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.468018    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.468267 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.468087    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wlckn\" (UniqueName: \"kubernetes.io/projected/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-kube-api-access-wlckn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.469147 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.469118    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.469190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.469125    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.476830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.476796    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlckn\" (UniqueName: \"kubernetes.io/projected/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-kube-api-access-wlckn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.595993 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.595802    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:25.747873 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.747805    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"]
Apr 16 16:15:25.752108 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:15:25.752061    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d6e34b_5c93_4afb_89da_b567a5bbb6b6.slice/crio-65123c3982c9eb25885d746b6fa47f1d117adc0da97a6f67e744e280208f3719 WatchSource:0}: Error finding container 65123c3982c9eb25885d746b6fa47f1d117adc0da97a6f67e744e280208f3719: Status 404 returned error can't find the container with id 65123c3982c9eb25885d746b6fa47f1d117adc0da97a6f67e744e280208f3719
Apr 16 16:15:25.848908 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.848746    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps" event={"ID":"58d6e34b-5c93-4afb-89da-b567a5bbb6b6","Type":"ContainerStarted","Data":"419a914d59af95cb3e1e71e54e5ad65bc5738de05e9079a0fd75df4ac7f58e2c"}
Apr 16 16:15:25.848908 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:25.848824    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps" event={"ID":"58d6e34b-5c93-4afb-89da-b567a5bbb6b6","Type":"ContainerStarted","Data":"65123c3982c9eb25885d746b6fa47f1d117adc0da97a6f67e744e280208f3719"}
Apr 16 16:15:26.855766 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:26.855573    2578 generic.go:358] "Generic (PLEG): container finished" podID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerID="419a914d59af95cb3e1e71e54e5ad65bc5738de05e9079a0fd75df4ac7f58e2c" exitCode=0
Apr 16 16:15:26.855766 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:26.855716    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps" event={"ID":"58d6e34b-5c93-4afb-89da-b567a5bbb6b6","Type":"ContainerDied","Data":"419a914d59af95cb3e1e71e54e5ad65bc5738de05e9079a0fd75df4ac7f58e2c"}
Apr 16 16:15:27.722149 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.722091    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"]
Apr 16 16:15:27.726899 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.726853    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:27.730430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.730388    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jr4jj\""
Apr 16 16:15:27.731139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.730938    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Apr 16 16:15:27.731642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.731379    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Apr 16 16:15:27.732909 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.732874    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"]
Apr 16 16:15:27.791279 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.791239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f7ccf8-be95-4b75-98ea-1b82f25b2e75-bound-sa-token\") pod \"cert-manager-cainjector-8966b78d4-vtpr2\" (UID: \"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75\") " pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:27.791493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.791289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztkb\" (UniqueName: \"kubernetes.io/projected/a5f7ccf8-be95-4b75-98ea-1b82f25b2e75-kube-api-access-6ztkb\") pod \"cert-manager-cainjector-8966b78d4-vtpr2\" (UID: \"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75\") " pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:27.892109 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.892054    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f7ccf8-be95-4b75-98ea-1b82f25b2e75-bound-sa-token\") pod \"cert-manager-cainjector-8966b78d4-vtpr2\" (UID: \"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75\") " pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:27.892109 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.892111    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztkb\" (UniqueName: \"kubernetes.io/projected/a5f7ccf8-be95-4b75-98ea-1b82f25b2e75-kube-api-access-6ztkb\") pod \"cert-manager-cainjector-8966b78d4-vtpr2\" (UID: \"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75\") " pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:27.902079 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.902039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5f7ccf8-be95-4b75-98ea-1b82f25b2e75-bound-sa-token\") pod \"cert-manager-cainjector-8966b78d4-vtpr2\" (UID: \"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75\") " pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:27.902316 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:27.902098    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztkb\" (UniqueName: \"kubernetes.io/projected/a5f7ccf8-be95-4b75-98ea-1b82f25b2e75-kube-api-access-6ztkb\") pod \"cert-manager-cainjector-8966b78d4-vtpr2\" (UID: \"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75\") " pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:28.051622 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:28.051481    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"
Apr 16 16:15:28.232521 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:28.229290    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-8966b78d4-vtpr2"]
Apr 16 16:15:28.232707 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:15:28.232512    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f7ccf8_be95_4b75_98ea_1b82f25b2e75.slice/crio-8629a95affe9e08dcbde5634be656ae8b7ebf1358d60ac2bfd664be3e3422c43 WatchSource:0}: Error finding container 8629a95affe9e08dcbde5634be656ae8b7ebf1358d60ac2bfd664be3e3422c43: Status 404 returned error can't find the container with id 8629a95affe9e08dcbde5634be656ae8b7ebf1358d60ac2bfd664be3e3422c43
Apr 16 16:15:28.875929 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:28.875846    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2" event={"ID":"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75","Type":"ContainerStarted","Data":"8629a95affe9e08dcbde5634be656ae8b7ebf1358d60ac2bfd664be3e3422c43"}
Apr 16 16:15:29.883480 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:29.883392    2578 generic.go:358] "Generic (PLEG): container finished" podID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerID="f7d67127dae0ae92f539de49cef103654f9ee9a4b022ac95f35b0b87116ee8de" exitCode=0
Apr 16 16:15:29.883480 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:29.883483    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps" event={"ID":"58d6e34b-5c93-4afb-89da-b567a5bbb6b6","Type":"ContainerDied","Data":"f7d67127dae0ae92f539de49cef103654f9ee9a4b022ac95f35b0b87116ee8de"}
Apr 16 16:15:30.892495 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:30.892397    2578 generic.go:358] "Generic (PLEG): container finished" podID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerID="876d280aa8562c1bb672272e3cdb735b7ad71163bc21010ec814a6afa64e8c30" exitCode=0
Apr 16 16:15:30.893167 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:30.892493    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps" event={"ID":"58d6e34b-5c93-4afb-89da-b567a5bbb6b6","Type":"ContainerDied","Data":"876d280aa8562c1bb672272e3cdb735b7ad71163bc21010ec814a6afa64e8c30"}
Apr 16 16:15:31.899500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:31.899339    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2" event={"ID":"a5f7ccf8-be95-4b75-98ea-1b82f25b2e75","Type":"ContainerStarted","Data":"d004c2914a09962a6966a2bd2a7affe3a9f8a15a1a81f2bdf51af9541848c310"}
Apr 16 16:15:31.914581 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:31.914475    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-8966b78d4-vtpr2" podStartSLOduration=1.7012835929999999 podStartE2EDuration="4.914426438s" podCreationTimestamp="2026-04-16 16:15:27 +0000 UTC" firstStartedPulling="2026-04-16 16:15:28.236613025 +0000 UTC m=+849.632697422" lastFinishedPulling="2026-04-16 16:15:31.449755866 +0000 UTC m=+852.845840267" observedRunningTime="2026-04-16 16:15:31.913524127 +0000 UTC m=+853.309608539" watchObservedRunningTime="2026-04-16 16:15:31.914426438 +0000 UTC m=+853.310510862"
Apr 16 16:15:32.058774 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.058742    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:32.143483 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.143395    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-util\") pod \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") "
Apr 16 16:15:32.143777 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.143542    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlckn\" (UniqueName: \"kubernetes.io/projected/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-kube-api-access-wlckn\") pod \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") "
Apr 16 16:15:32.143777 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.143617    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-bundle\") pod \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\" (UID: \"58d6e34b-5c93-4afb-89da-b567a5bbb6b6\") "
Apr 16 16:15:32.144071 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.144042    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-bundle" (OuterVolumeSpecName: "bundle") pod "58d6e34b-5c93-4afb-89da-b567a5bbb6b6" (UID: "58d6e34b-5c93-4afb-89da-b567a5bbb6b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:15:32.146479 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.146431    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-kube-api-access-wlckn" (OuterVolumeSpecName: "kube-api-access-wlckn") pod "58d6e34b-5c93-4afb-89da-b567a5bbb6b6" (UID: "58d6e34b-5c93-4afb-89da-b567a5bbb6b6"). InnerVolumeSpecName "kube-api-access-wlckn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:15:32.148793 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.148761    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-util" (OuterVolumeSpecName: "util") pod "58d6e34b-5c93-4afb-89da-b567a5bbb6b6" (UID: "58d6e34b-5c93-4afb-89da-b567a5bbb6b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:15:32.245257 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.245207    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlckn\" (UniqueName: \"kubernetes.io/projected/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-kube-api-access-wlckn\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:32.245257 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.245247    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:32.245257 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.245265    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58d6e34b-5c93-4afb-89da-b567a5bbb6b6-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:32.904861 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.904803    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps" event={"ID":"58d6e34b-5c93-4afb-89da-b567a5bbb6b6","Type":"ContainerDied","Data":"65123c3982c9eb25885d746b6fa47f1d117adc0da97a6f67e744e280208f3719"}
Apr 16 16:15:32.904861 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.904837    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4v2ps"
Apr 16 16:15:32.904861 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:32.904853    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65123c3982c9eb25885d746b6fa47f1d117adc0da97a6f67e744e280208f3719"
Apr 16 16:15:39.928970 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.928936    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"]
Apr 16 16:15:39.929473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.929316    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerName="util"
Apr 16 16:15:39.929473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.929329    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerName="util"
Apr 16 16:15:39.929473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.929338    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerName="extract"
Apr 16 16:15:39.929473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.929344    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerName="extract"
Apr 16 16:15:39.929473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.929354    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerName="pull"
Apr 16 16:15:39.929473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.929360    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerName="pull"
Apr 16 16:15:39.929473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.929421    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="58d6e34b-5c93-4afb-89da-b567a5bbb6b6" containerName="extract"
Apr 16 16:15:39.934595 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.934574    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:39.936968 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.936942    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Apr 16 16:15:39.937092 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.936949    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wwmq5\""
Apr 16 16:15:39.938024 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.938005    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Apr 16 16:15:39.941582 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:39.941559    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"]
Apr 16 16:15:40.020301 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.020261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvn84\" (UniqueName: \"kubernetes.io/projected/9c839d5e-6305-438c-b620-79acbbf1760d-kube-api-access-zvn84\") pod \"openshift-lws-operator-bfc7f696d-kgvsc\" (UID: \"9c839d5e-6305-438c-b620-79acbbf1760d\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:40.020538 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.020362    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c839d5e-6305-438c-b620-79acbbf1760d-tmp\") pod \"openshift-lws-operator-bfc7f696d-kgvsc\" (UID: \"9c839d5e-6305-438c-b620-79acbbf1760d\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:40.121252 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.121210    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c839d5e-6305-438c-b620-79acbbf1760d-tmp\") pod \"openshift-lws-operator-bfc7f696d-kgvsc\" (UID: \"9c839d5e-6305-438c-b620-79acbbf1760d\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:40.121513 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.121273    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zvn84\" (UniqueName: \"kubernetes.io/projected/9c839d5e-6305-438c-b620-79acbbf1760d-kube-api-access-zvn84\") pod \"openshift-lws-operator-bfc7f696d-kgvsc\" (UID: \"9c839d5e-6305-438c-b620-79acbbf1760d\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:40.121646 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.121622    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c839d5e-6305-438c-b620-79acbbf1760d-tmp\") pod \"openshift-lws-operator-bfc7f696d-kgvsc\" (UID: \"9c839d5e-6305-438c-b620-79acbbf1760d\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:40.130561 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.130533    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvn84\" (UniqueName: \"kubernetes.io/projected/9c839d5e-6305-438c-b620-79acbbf1760d-kube-api-access-zvn84\") pod \"openshift-lws-operator-bfc7f696d-kgvsc\" (UID: \"9c839d5e-6305-438c-b620-79acbbf1760d\") " pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:40.244612 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.244571    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"
Apr 16 16:15:40.371397 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.371290    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc"]
Apr 16 16:15:40.373987 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:15:40.373952    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c839d5e_6305_438c_b620_79acbbf1760d.slice/crio-b2db4f2e6c81aac2b23b6a24e8fc98b25f27432f98f2da08531bea0ce0ec3bad WatchSource:0}: Error finding container b2db4f2e6c81aac2b23b6a24e8fc98b25f27432f98f2da08531bea0ce0ec3bad: Status 404 returned error can't find the container with id b2db4f2e6c81aac2b23b6a24e8fc98b25f27432f98f2da08531bea0ce0ec3bad
Apr 16 16:15:40.937135 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:40.937093    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc" event={"ID":"9c839d5e-6305-438c-b620-79acbbf1760d","Type":"ContainerStarted","Data":"b2db4f2e6c81aac2b23b6a24e8fc98b25f27432f98f2da08531bea0ce0ec3bad"}
Apr 16 16:15:42.946615 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:42.946578    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc" event={"ID":"9c839d5e-6305-438c-b620-79acbbf1760d","Type":"ContainerStarted","Data":"716d3f82541dc54242d3e430081bab9a0968ee74a7a8a7365ade36a9a2d3113c"}
Apr 16 16:15:42.960819 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:42.960762    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-bfc7f696d-kgvsc" podStartSLOduration=2.124162273 podStartE2EDuration="3.960743505s" podCreationTimestamp="2026-04-16 16:15:39 +0000 UTC" firstStartedPulling="2026-04-16 16:15:40.375520887 +0000 UTC m=+861.771605285" lastFinishedPulling="2026-04-16 16:15:42.212102115 +0000 UTC m=+863.608186517" observedRunningTime="2026-04-16 16:15:42.959901192 +0000 UTC m=+864.355985624" watchObservedRunningTime="2026-04-16 16:15:42.960743505 +0000 UTC m=+864.356827928"
Apr 16 16:15:53.251183 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.251146    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"]
Apr 16 16:15:53.254838 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.254819    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.257221 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.257194    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 16 16:15:53.257348 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.257198    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 16 16:15:53.257348 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.257238    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9wlr7\""
Apr 16 16:15:53.262316 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.262295    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"]
Apr 16 16:15:53.341506 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.341478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.341616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.341540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpwk\" (UniqueName: \"kubernetes.io/projected/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-kube-api-access-5kpwk\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.341616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.341598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.442994 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.442951    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.443179 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.443011    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpwk\" (UniqueName: \"kubernetes.io/projected/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-kube-api-access-5kpwk\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.443179 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.443037    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.443360 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.443345    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-util\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.443428 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.443404    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-bundle\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.452232 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.452207    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpwk\" (UniqueName: \"kubernetes.io/projected/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-kube-api-access-5kpwk\") pod \"4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") " pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.592937 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.592852    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:53.715893 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.715860    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"]
Apr 16 16:15:53.718269 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:15:53.718241    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8245dc94_ee8e_4d87_9d6e_027ec09daaaa.slice/crio-fff959357b65248f68ae71bd0a29e37a6043248b1c1d5aeb867a5a6be1605c71 WatchSource:0}: Error finding container fff959357b65248f68ae71bd0a29e37a6043248b1c1d5aeb867a5a6be1605c71: Status 404 returned error can't find the container with id fff959357b65248f68ae71bd0a29e37a6043248b1c1d5aeb867a5a6be1605c71
Apr 16 16:15:53.984836 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.984796    2578 generic.go:358] "Generic (PLEG): container finished" podID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerID="4f288459e362183e472928dfb1e2aadda27d11ab04e321981f963af858a49163" exitCode=0
Apr 16 16:15:53.984988 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.984879    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn" event={"ID":"8245dc94-ee8e-4d87-9d6e-027ec09daaaa","Type":"ContainerDied","Data":"4f288459e362183e472928dfb1e2aadda27d11ab04e321981f963af858a49163"}
Apr 16 16:15:53.984988 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:53.984913    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn" event={"ID":"8245dc94-ee8e-4d87-9d6e-027ec09daaaa","Type":"ContainerStarted","Data":"fff959357b65248f68ae71bd0a29e37a6043248b1c1d5aeb867a5a6be1605c71"}
Apr 16 16:15:55.995681 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:55.995641    2578 generic.go:358] "Generic (PLEG): container finished" podID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerID="b082f8ba4ada3e78ae261e3f781bc74e3518a91bd740a7f31ca572ba4a227ed2" exitCode=0
Apr 16 16:15:55.996103 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:55.995726    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn" event={"ID":"8245dc94-ee8e-4d87-9d6e-027ec09daaaa","Type":"ContainerDied","Data":"b082f8ba4ada3e78ae261e3f781bc74e3518a91bd740a7f31ca572ba4a227ed2"}
Apr 16 16:15:57.001961 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:57.001920    2578 generic.go:358] "Generic (PLEG): container finished" podID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerID="5ea0cbd51f5fe8b505bd18880dbcb44156f114e89020430901309210651cdc9b" exitCode=0
Apr 16 16:15:57.002371 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:57.001992    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn" event={"ID":"8245dc94-ee8e-4d87-9d6e-027ec09daaaa","Type":"ContainerDied","Data":"5ea0cbd51f5fe8b505bd18880dbcb44156f114e89020430901309210651cdc9b"}
Apr 16 16:15:58.136614 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.136592    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:58.281752 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.281665    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kpwk\" (UniqueName: \"kubernetes.io/projected/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-kube-api-access-5kpwk\") pod \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") "
Apr 16 16:15:58.281752 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.281720    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-util\") pod \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") "
Apr 16 16:15:58.281965 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.281829    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-bundle\") pod \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\" (UID: \"8245dc94-ee8e-4d87-9d6e-027ec09daaaa\") "
Apr 16 16:15:58.282665 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.282632    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-bundle" (OuterVolumeSpecName: "bundle") pod "8245dc94-ee8e-4d87-9d6e-027ec09daaaa" (UID: "8245dc94-ee8e-4d87-9d6e-027ec09daaaa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:15:58.283975 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.283951    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-kube-api-access-5kpwk" (OuterVolumeSpecName: "kube-api-access-5kpwk") pod "8245dc94-ee8e-4d87-9d6e-027ec09daaaa" (UID: "8245dc94-ee8e-4d87-9d6e-027ec09daaaa"). InnerVolumeSpecName "kube-api-access-5kpwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:15:58.289968 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.289924    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-util" (OuterVolumeSpecName: "util") pod "8245dc94-ee8e-4d87-9d6e-027ec09daaaa" (UID: "8245dc94-ee8e-4d87-9d6e-027ec09daaaa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:15:58.382983 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.382944    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:58.382983 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.382978    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kpwk\" (UniqueName: \"kubernetes.io/projected/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-kube-api-access-5kpwk\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:58.382983 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:58.382989    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8245dc94-ee8e-4d87-9d6e-027ec09daaaa-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:15:59.017355 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:59.017330    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn"
Apr 16 16:15:59.017552 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:59.017323    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/4c892a6d2d8a57acbc49427d3a7d24e253bab25be2c607ef405d2c4835np8qn" event={"ID":"8245dc94-ee8e-4d87-9d6e-027ec09daaaa","Type":"ContainerDied","Data":"fff959357b65248f68ae71bd0a29e37a6043248b1c1d5aeb867a5a6be1605c71"}
Apr 16 16:15:59.017552 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:15:59.017435    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff959357b65248f68ae71bd0a29e37a6043248b1c1d5aeb867a5a6be1605c71"
Apr 16 16:16:08.004390 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004355    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"]
Apr 16 16:16:08.004874 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004859    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerName="pull"
Apr 16 16:16:08.004952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004878    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerName="pull"
Apr 16 16:16:08.004952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004897    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerName="extract"
Apr 16 16:16:08.004952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004905    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerName="extract"
Apr 16 16:16:08.004952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004917    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerName="util"
Apr 16 16:16:08.004952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004925    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerName="util"
Apr 16 16:16:08.005161 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.004993    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="8245dc94-ee8e-4d87-9d6e-027ec09daaaa" containerName="extract"
Apr 16 16:16:08.008424 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.008402    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.011100 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.011083    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 16 16:16:08.012276 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.012093    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 16 16:16:08.012276 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.012103    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9wlr7\""
Apr 16 16:16:08.014309 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.014284    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"]
Apr 16 16:16:08.160939 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.160909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-util\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.161071 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.160943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmc8w\" (UniqueName: \"kubernetes.io/projected/e8aecec1-b76b-4098-a1a0-61c716b79aef-kube-api-access-lmc8w\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.161071 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.160970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-bundle\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.261737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.261678    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-util\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.261737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.261709    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lmc8w\" (UniqueName: \"kubernetes.io/projected/e8aecec1-b76b-4098-a1a0-61c716b79aef-kube-api-access-lmc8w\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.261737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.261731    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-bundle\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.262013 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.261995    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-util\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.262066 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.262029    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-bundle\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.271274 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.271256    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmc8w\" (UniqueName: \"kubernetes.io/projected/e8aecec1-b76b-4098-a1a0-61c716b79aef-kube-api-access-lmc8w\") pod \"d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") " pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.320170 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.320146    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:08.440313 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:08.440290    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"]
Apr 16 16:16:08.442517 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:08.442485    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8aecec1_b76b_4098_a1a0_61c716b79aef.slice/crio-5702849aa67b0922f4a89cc38f308b0ad73157a05ac0ea2e002015e231251e90 WatchSource:0}: Error finding container 5702849aa67b0922f4a89cc38f308b0ad73157a05ac0ea2e002015e231251e90: Status 404 returned error can't find the container with id 5702849aa67b0922f4a89cc38f308b0ad73157a05ac0ea2e002015e231251e90
Apr 16 16:16:09.058988 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:09.058956    2578 generic.go:358] "Generic (PLEG): container finished" podID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerID="00adcbf4d11944811afaab8dd4156bcd547f68342d772f040b075b0e0eb0de57" exitCode=0
Apr 16 16:16:09.059359 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:09.059017    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b" event={"ID":"e8aecec1-b76b-4098-a1a0-61c716b79aef","Type":"ContainerDied","Data":"00adcbf4d11944811afaab8dd4156bcd547f68342d772f040b075b0e0eb0de57"}
Apr 16 16:16:09.059359 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:09.059040    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b" event={"ID":"e8aecec1-b76b-4098-a1a0-61c716b79aef","Type":"ContainerStarted","Data":"5702849aa67b0922f4a89cc38f308b0ad73157a05ac0ea2e002015e231251e90"}
Apr 16 16:16:10.268387 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.268362    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"]
Apr 16 16:16:10.271565 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.271551    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.274078 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.274055    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-operators\"/\"servicemesh-operator3-dockercfg-7cnfl\""
Apr 16 16:16:10.274620 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.274602    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"openshift-service-ca.crt\""
Apr 16 16:16:10.274715 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.274623    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-operators\"/\"kube-root-ca.crt\""
Apr 16 16:16:10.281561 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.281539    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"]
Apr 16 16:16:10.376490 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.376415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/3209de80-b30b-4dd4-ab5e-bed8815e4362-operator-config\") pod \"servicemesh-operator3-55f49c5f94-kvrf9\" (UID: \"3209de80-b30b-4dd4-ab5e-bed8815e4362\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.376490 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.376442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnznr\" (UniqueName: \"kubernetes.io/projected/3209de80-b30b-4dd4-ab5e-bed8815e4362-kube-api-access-wnznr\") pod \"servicemesh-operator3-55f49c5f94-kvrf9\" (UID: \"3209de80-b30b-4dd4-ab5e-bed8815e4362\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.477533 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.477507    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/3209de80-b30b-4dd4-ab5e-bed8815e4362-operator-config\") pod \"servicemesh-operator3-55f49c5f94-kvrf9\" (UID: \"3209de80-b30b-4dd4-ab5e-bed8815e4362\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.477641 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.477540    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wnznr\" (UniqueName: \"kubernetes.io/projected/3209de80-b30b-4dd4-ab5e-bed8815e4362-kube-api-access-wnznr\") pod \"servicemesh-operator3-55f49c5f94-kvrf9\" (UID: \"3209de80-b30b-4dd4-ab5e-bed8815e4362\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.479976 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.479956    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"operator-config\" (UniqueName: \"kubernetes.io/downward-api/3209de80-b30b-4dd4-ab5e-bed8815e4362-operator-config\") pod \"servicemesh-operator3-55f49c5f94-kvrf9\" (UID: \"3209de80-b30b-4dd4-ab5e-bed8815e4362\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.488196 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.488175    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnznr\" (UniqueName: \"kubernetes.io/projected/3209de80-b30b-4dd4-ab5e-bed8815e4362-kube-api-access-wnznr\") pod \"servicemesh-operator3-55f49c5f94-kvrf9\" (UID: \"3209de80-b30b-4dd4-ab5e-bed8815e4362\") " pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.580628 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.580605    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:10.705204 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:10.705174    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"]
Apr 16 16:16:10.708346 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:10.708300    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3209de80_b30b_4dd4_ab5e_bed8815e4362.slice/crio-9b4a113eb405507b15831b36a4afc1ff7ef7be857e2182fb2c1f2d2742df4de5 WatchSource:0}: Error finding container 9b4a113eb405507b15831b36a4afc1ff7ef7be857e2182fb2c1f2d2742df4de5: Status 404 returned error can't find the container with id 9b4a113eb405507b15831b36a4afc1ff7ef7be857e2182fb2c1f2d2742df4de5
Apr 16 16:16:11.067145 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:11.067104    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9" event={"ID":"3209de80-b30b-4dd4-ab5e-bed8815e4362","Type":"ContainerStarted","Data":"9b4a113eb405507b15831b36a4afc1ff7ef7be857e2182fb2c1f2d2742df4de5"}
Apr 16 16:16:11.068760 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:11.068731    2578 generic.go:358] "Generic (PLEG): container finished" podID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerID="f451428511d933d854dd6030e1e35249b8e1f5903f45359f1f0022f7f91a8d50" exitCode=0
Apr 16 16:16:11.068889 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:11.068808    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b" event={"ID":"e8aecec1-b76b-4098-a1a0-61c716b79aef","Type":"ContainerDied","Data":"f451428511d933d854dd6030e1e35249b8e1f5903f45359f1f0022f7f91a8d50"}
Apr 16 16:16:12.075280 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:12.075242    2578 generic.go:358] "Generic (PLEG): container finished" podID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerID="be35c8c476d908e1608b58cad083ae091910d36e4f83a31d19294c7b84b394f2" exitCode=0
Apr 16 16:16:12.075670 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:12.075323    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b" event={"ID":"e8aecec1-b76b-4098-a1a0-61c716b79aef","Type":"ContainerDied","Data":"be35c8c476d908e1608b58cad083ae091910d36e4f83a31d19294c7b84b394f2"}
Apr 16 16:16:13.210589 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.210567    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:13.298664 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.298635    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmc8w\" (UniqueName: \"kubernetes.io/projected/e8aecec1-b76b-4098-a1a0-61c716b79aef-kube-api-access-lmc8w\") pod \"e8aecec1-b76b-4098-a1a0-61c716b79aef\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") "
Apr 16 16:16:13.298840 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.298688    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-bundle\") pod \"e8aecec1-b76b-4098-a1a0-61c716b79aef\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") "
Apr 16 16:16:13.298840 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.298734    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-util\") pod \"e8aecec1-b76b-4098-a1a0-61c716b79aef\" (UID: \"e8aecec1-b76b-4098-a1a0-61c716b79aef\") "
Apr 16 16:16:13.299536 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.299511    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-bundle" (OuterVolumeSpecName: "bundle") pod "e8aecec1-b76b-4098-a1a0-61c716b79aef" (UID: "e8aecec1-b76b-4098-a1a0-61c716b79aef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:13.300894 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.300870    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8aecec1-b76b-4098-a1a0-61c716b79aef-kube-api-access-lmc8w" (OuterVolumeSpecName: "kube-api-access-lmc8w") pod "e8aecec1-b76b-4098-a1a0-61c716b79aef" (UID: "e8aecec1-b76b-4098-a1a0-61c716b79aef"). InnerVolumeSpecName "kube-api-access-lmc8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:16:13.304403 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.304373    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-util" (OuterVolumeSpecName: "util") pod "e8aecec1-b76b-4098-a1a0-61c716b79aef" (UID: "e8aecec1-b76b-4098-a1a0-61c716b79aef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:13.399911 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.399843    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:13.399911 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.399867    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8aecec1-b76b-4098-a1a0-61c716b79aef-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:13.399911 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:13.399879    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmc8w\" (UniqueName: \"kubernetes.io/projected/e8aecec1-b76b-4098-a1a0-61c716b79aef-kube-api-access-lmc8w\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:14.084813 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:14.084774    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b" event={"ID":"e8aecec1-b76b-4098-a1a0-61c716b79aef","Type":"ContainerDied","Data":"5702849aa67b0922f4a89cc38f308b0ad73157a05ac0ea2e002015e231251e90"}
Apr 16 16:16:14.084813 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:14.084813    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5702849aa67b0922f4a89cc38f308b0ad73157a05ac0ea2e002015e231251e90"
Apr 16 16:16:14.085053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:14.084827    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d813cbca2f2d4ae5bb9c3e9ca6bc9dc97fa22f4f10cc797dd3b2c1f0c2cl22b"
Apr 16 16:16:16.092978 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:16.092923    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9" event={"ID":"3209de80-b30b-4dd4-ab5e-bed8815e4362","Type":"ContainerStarted","Data":"e00cb607bdd816705cd9911041dfcc0ed947975e867ebb4c909a88b8959fb55e"}
Apr 16 16:16:16.093374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:16.093011    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:16.114696 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:16.114651    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9" podStartSLOduration=1.7434695690000002 podStartE2EDuration="6.114639308s" podCreationTimestamp="2026-04-16 16:16:10 +0000 UTC" firstStartedPulling="2026-04-16 16:16:10.711093585 +0000 UTC m=+892.107177989" lastFinishedPulling="2026-04-16 16:16:15.082263328 +0000 UTC m=+896.478347728" observedRunningTime="2026-04-16 16:16:16.111674294 +0000 UTC m=+897.507758715" watchObservedRunningTime="2026-04-16 16:16:16.114639308 +0000 UTC m=+897.510723728"
Apr 16 16:16:18.188121 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188084    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"]
Apr 16 16:16:18.188576 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188560    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerName="extract"
Apr 16 16:16:18.188635 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188579    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerName="extract"
Apr 16 16:16:18.188635 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188614    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerName="util"
Apr 16 16:16:18.188635 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188622    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerName="util"
Apr 16 16:16:18.188726 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188641    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerName="pull"
Apr 16 16:16:18.188726 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188649    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerName="pull"
Apr 16 16:16:18.188783 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.188725    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8aecec1-b76b-4098-a1a0-61c716b79aef" containerName="extract"
Apr 16 16:16:18.192260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.192243    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.195522 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.195498    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k7fzl\""
Apr 16 16:16:18.195653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.195501    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Apr 16 16:16:18.195653 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.195540    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Apr 16 16:16:18.195945 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.195909    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"istio-ca-root-cert\""
Apr 16 16:16:18.195945 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.195940    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Apr 16 16:16:18.196062 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.196019    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Apr 16 16:16:18.196634 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.196619    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Apr 16 16:16:18.202177 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.202142    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"]
Apr 16 16:16:18.234996 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.234965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbvq\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-kube-api-access-sgbvq\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.234996 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.234997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/584f000a-b973-44a3-900d-b8473287084c-local-certs\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.235166 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.235013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-istio-token\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.235166 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.235042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.235166 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.235057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/584f000a-b973-44a3-900d-b8473287084c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.235320 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.235172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.235320 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.235220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-cacerts\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.336137 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.336110    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbvq\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-kube-api-access-sgbvq\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.336233 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.336142    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/584f000a-b973-44a3-900d-b8473287084c-local-certs\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.336233 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.336159    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-istio-token\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.336233 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.336186    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.336341 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.336300    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/584f000a-b973-44a3-900d-b8473287084c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.336389 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.336373    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.336441 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.336414    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-cacerts\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.337067 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.337018    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/584f000a-b973-44a3-900d-b8473287084c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.339053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.338983    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/584f000a-b973-44a3-900d-b8473287084c-local-certs\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.339322 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.339301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-cacerts\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.339371 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.339333    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.339406 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.339394    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.343629 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.343608    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbvq\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-kube-api-access-sgbvq\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.343734 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.343666    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-istio-token\") pod \"istiod-openshift-gateway-7cd77c7ffd-pq77l\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") " pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.502121 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.502084    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:18.663195 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:18.663172    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"]
Apr 16 16:16:18.665210 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:18.665165    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584f000a_b973_44a3_900d_b8473287084c.slice/crio-c2fdb73ed2abbb8ef0396db508076015d8a763486615e7a202da8b8b02edad4d WatchSource:0}: Error finding container c2fdb73ed2abbb8ef0396db508076015d8a763486615e7a202da8b8b02edad4d: Status 404 returned error can't find the container with id c2fdb73ed2abbb8ef0396db508076015d8a763486615e7a202da8b8b02edad4d
Apr 16 16:16:19.077132 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:19.077106    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:16:19.077325 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:19.077258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:16:19.107254 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:19.107222    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l" event={"ID":"584f000a-b973-44a3-900d-b8473287084c","Type":"ContainerStarted","Data":"c2fdb73ed2abbb8ef0396db508076015d8a763486615e7a202da8b8b02edad4d"}
Apr 16 16:16:21.175529 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:21.175496    2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Apr 16 16:16:21.175772 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:21.175561    2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Apr 16 16:16:22.121007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:22.120970    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l" event={"ID":"584f000a-b973-44a3-900d-b8473287084c","Type":"ContainerStarted","Data":"7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587"}
Apr 16 16:16:22.121183 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:22.121020    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:22.122891 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:22.122862    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7cd77c7ffd-pq77l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Apr 16 16:16:22.123019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:22.122912    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l" podUID="584f000a-b973-44a3-900d-b8473287084c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:16:22.141614 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:22.141562    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l" podStartSLOduration=1.633925177 podStartE2EDuration="4.141546085s" podCreationTimestamp="2026-04-16 16:16:18 +0000 UTC" firstStartedPulling="2026-04-16 16:16:18.667633908 +0000 UTC m=+900.063718306" lastFinishedPulling="2026-04-16 16:16:21.175254816 +0000 UTC m=+902.571339214" observedRunningTime="2026-04-16 16:16:22.1405812 +0000 UTC m=+903.536665624" watchObservedRunningTime="2026-04-16 16:16:22.141546085 +0000 UTC m=+903.537630506"
Apr 16 16:16:23.129789 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:23.129755    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:16:27.099137 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:27.099105    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-operators/servicemesh-operator3-55f49c5f94-kvrf9"
Apr 16 16:16:32.265851 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.265816    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"]
Apr 16 16:16:32.270801 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.270785    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.274015 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.273993    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Apr 16 16:16:32.274148 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.274056    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9wlr7\""
Apr 16 16:16:32.274148 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.274066    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Apr 16 16:16:32.276815 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.276792    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"]
Apr 16 16:16:32.351748 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.351725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjr56\" (UniqueName: \"kubernetes.io/projected/74be648d-3de4-48fa-bcbb-19ce18467ca2-kube-api-access-vjr56\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.351903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.351770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.351903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.351810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.366148 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.366119    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"]
Apr 16 16:16:32.369742 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.369728    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.376272 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.376234    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"]
Apr 16 16:16:32.452697 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.452657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.452820 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.452706    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vjr56\" (UniqueName: \"kubernetes.io/projected/74be648d-3de4-48fa-bcbb-19ce18467ca2-kube-api-access-vjr56\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.452820 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.452785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7tn\" (UniqueName: \"kubernetes.io/projected/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-kube-api-access-qr7tn\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.452940 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.452825    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.452940 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.452854    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.452940 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.452917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.453151 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.453136    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-util\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.453198 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.453166    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-bundle\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.460433 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.460410    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjr56\" (UniqueName: \"kubernetes.io/projected/74be648d-3de4-48fa-bcbb-19ce18467ca2-kube-api-access-vjr56\") pod \"309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") " pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.465985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.465965    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"]
Apr 16 16:16:32.469338 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.469322    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.477178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.477156    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"]
Apr 16 16:16:32.553570 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553508    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.553570 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553551    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.553719 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.553719 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553610    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7tn\" (UniqueName: \"kubernetes.io/projected/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-kube-api-access-qr7tn\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.553719 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.553719 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xd85\" (UniqueName: \"kubernetes.io/projected/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-kube-api-access-6xd85\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.553910 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553847    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-bundle\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.553910 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.553880    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-util\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.573224 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.573199    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"]
Apr 16 16:16:32.574885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.574867    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7tn\" (UniqueName: \"kubernetes.io/projected/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-kube-api-access-qr7tn\") pod \"ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") " pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.576914 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.576895    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.579923 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.579899    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"]
Apr 16 16:16:32.580151 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.580133    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:32.654848 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.654791    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.654996 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.654848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-bundle\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.654996 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.654945    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6xd85\" (UniqueName: \"kubernetes.io/projected/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-kube-api-access-6xd85\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.655357 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.655006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-util\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.655357 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.655036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtf4\" (UniqueName: \"kubernetes.io/projected/133b1df4-7046-4d19-8f72-860dc14f9d28-kube-api-access-lbtf4\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.655357 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.655112    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.655616 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.655478    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-bundle\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.655768 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.655676    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-util\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.662911 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.662888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xd85\" (UniqueName: \"kubernetes.io/projected/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-kube-api-access-6xd85\") pod \"d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") " pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.679756 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.679735    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:32.704563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.704538    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"]
Apr 16 16:16:32.706807 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:32.706777    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74be648d_3de4_48fa_bcbb_19ce18467ca2.slice/crio-f64a5f0099d34f4a782cfc7af20d079fb320cb52db623ea12d7398abfc3bc9b4 WatchSource:0}: Error finding container f64a5f0099d34f4a782cfc7af20d079fb320cb52db623ea12d7398abfc3bc9b4: Status 404 returned error can't find the container with id f64a5f0099d34f4a782cfc7af20d079fb320cb52db623ea12d7398abfc3bc9b4
Apr 16 16:16:32.756595 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.756569    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-bundle\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.756711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.756693    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-util\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.756758 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.756733    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtf4\" (UniqueName: \"kubernetes.io/projected/133b1df4-7046-4d19-8f72-860dc14f9d28-kube-api-access-lbtf4\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.756935 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.756915    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-bundle\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.757053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.757031    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-util\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.765908 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.765887    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtf4\" (UniqueName: \"kubernetes.io/projected/133b1df4-7046-4d19-8f72-860dc14f9d28-kube-api-access-lbtf4\") pod \"5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") " pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.778998 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.778686    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:32.809198 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.809167    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"]
Apr 16 16:16:32.809972 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:32.809942    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e5d39a_e3bd_4152_a5b9_2ee520b92b42.slice/crio-65dd312969820259b3b155df962117449842cc45e2aa073bfa201fca6d1b0bbe WatchSource:0}: Error finding container 65dd312969820259b3b155df962117449842cc45e2aa073bfa201fca6d1b0bbe: Status 404 returned error can't find the container with id 65dd312969820259b3b155df962117449842cc45e2aa073bfa201fca6d1b0bbe
Apr 16 16:16:32.904938 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.904906    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:32.914173 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:32.914145    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"]
Apr 16 16:16:32.914514 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:32.914488    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbae3ae5_a40d_4bdb_bced_b644817a7fe5.slice/crio-632a1c8e68e500816874b26c42f116baa27362c9976c59b08e94721ad696cfe5 WatchSource:0}: Error finding container 632a1c8e68e500816874b26c42f116baa27362c9976c59b08e94721ad696cfe5: Status 404 returned error can't find the container with id 632a1c8e68e500816874b26c42f116baa27362c9976c59b08e94721ad696cfe5
Apr 16 16:16:33.028615 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.028592    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"]
Apr 16 16:16:33.030203 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:33.030179    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133b1df4_7046_4d19_8f72_860dc14f9d28.slice/crio-a7dbd725cb91152c7aa0fc834626a4a37afa94225d51f5c06928240f93c8c3b4 WatchSource:0}: Error finding container a7dbd725cb91152c7aa0fc834626a4a37afa94225d51f5c06928240f93c8c3b4: Status 404 returned error can't find the container with id a7dbd725cb91152c7aa0fc834626a4a37afa94225d51f5c06928240f93c8c3b4
Apr 16 16:16:33.166991 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.166965    2578 generic.go:358] "Generic (PLEG): container finished" podID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerID="01db3c61f46800840e724fa0232f1444af796b9f33ce7313504eaa44517cf22c" exitCode=0
Apr 16 16:16:33.167119 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.167054    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f" event={"ID":"cbae3ae5-a40d-4bdb-bced-b644817a7fe5","Type":"ContainerDied","Data":"01db3c61f46800840e724fa0232f1444af796b9f33ce7313504eaa44517cf22c"}
Apr 16 16:16:33.167119 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.167102    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f" event={"ID":"cbae3ae5-a40d-4bdb-bced-b644817a7fe5","Type":"ContainerStarted","Data":"632a1c8e68e500816874b26c42f116baa27362c9976c59b08e94721ad696cfe5"}
Apr 16 16:16:33.168716 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.168687    2578 generic.go:358] "Generic (PLEG): container finished" podID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerID="0455f443eeff156b096b3f01957b7a4c417dfab1d8affeffa09c44595bf77a71" exitCode=0
Apr 16 16:16:33.168804 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.168765    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg" event={"ID":"54e5d39a-e3bd-4152-a5b9-2ee520b92b42","Type":"ContainerDied","Data":"0455f443eeff156b096b3f01957b7a4c417dfab1d8affeffa09c44595bf77a71"}
Apr 16 16:16:33.168873 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.168808    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg" event={"ID":"54e5d39a-e3bd-4152-a5b9-2ee520b92b42","Type":"ContainerStarted","Data":"65dd312969820259b3b155df962117449842cc45e2aa073bfa201fca6d1b0bbe"}
Apr 16 16:16:33.170307 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.170287    2578 generic.go:358] "Generic (PLEG): container finished" podID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerID="6ea4dca091cff158a33476d78b584f148f86a8ea9ce8853871c65b259f911acb" exitCode=0
Apr 16 16:16:33.170394 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.170358    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb" event={"ID":"133b1df4-7046-4d19-8f72-860dc14f9d28","Type":"ContainerDied","Data":"6ea4dca091cff158a33476d78b584f148f86a8ea9ce8853871c65b259f911acb"}
Apr 16 16:16:33.170394 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.170381    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb" event={"ID":"133b1df4-7046-4d19-8f72-860dc14f9d28","Type":"ContainerStarted","Data":"a7dbd725cb91152c7aa0fc834626a4a37afa94225d51f5c06928240f93c8c3b4"}
Apr 16 16:16:33.172075 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.172056    2578 generic.go:358] "Generic (PLEG): container finished" podID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerID="33905fb039ba006bd508f0130cc563ea677089c52ca1ad2827fcd5911dd06ca6" exitCode=0
Apr 16 16:16:33.172154 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.172122    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8" event={"ID":"74be648d-3de4-48fa-bcbb-19ce18467ca2","Type":"ContainerDied","Data":"33905fb039ba006bd508f0130cc563ea677089c52ca1ad2827fcd5911dd06ca6"}
Apr 16 16:16:33.172154 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:33.172146    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8" event={"ID":"74be648d-3de4-48fa-bcbb-19ce18467ca2","Type":"ContainerStarted","Data":"f64a5f0099d34f4a782cfc7af20d079fb320cb52db623ea12d7398abfc3bc9b4"}
Apr 16 16:16:34.178415 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:34.178386    2578 generic.go:358] "Generic (PLEG): container finished" podID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerID="4180fd4706ff7ea7c33ee2790936c5e4e9a6f8b01fdc2f8fe3ad624dcaafe5cb" exitCode=0
Apr 16 16:16:34.178859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:34.178547    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f" event={"ID":"cbae3ae5-a40d-4bdb-bced-b644817a7fe5","Type":"ContainerDied","Data":"4180fd4706ff7ea7c33ee2790936c5e4e9a6f8b01fdc2f8fe3ad624dcaafe5cb"}
Apr 16 16:16:34.180165 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:34.180143    2578 generic.go:358] "Generic (PLEG): container finished" podID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerID="9f1fa9e705a924c4d5194114ad261737776ae76f63b8679bdd4306c1528678fe" exitCode=0
Apr 16 16:16:34.180260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:34.180222    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg" event={"ID":"54e5d39a-e3bd-4152-a5b9-2ee520b92b42","Type":"ContainerDied","Data":"9f1fa9e705a924c4d5194114ad261737776ae76f63b8679bdd4306c1528678fe"}
Apr 16 16:16:34.181759 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:34.181737    2578 generic.go:358] "Generic (PLEG): container finished" podID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerID="357b488223e9f480d646be647e382411525e46f2e087bd63e34d1edbd53ca665" exitCode=0
Apr 16 16:16:34.181828 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:34.181783    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8" event={"ID":"74be648d-3de4-48fa-bcbb-19ce18467ca2","Type":"ContainerDied","Data":"357b488223e9f480d646be647e382411525e46f2e087bd63e34d1edbd53ca665"}
Apr 16 16:16:35.187264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.187229    2578 generic.go:358] "Generic (PLEG): container finished" podID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerID="7600d876981a90174fbeefc2146a03d5e3665d5803119a27ea5927d696d38043" exitCode=0
Apr 16 16:16:35.187722 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.187323    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg" event={"ID":"54e5d39a-e3bd-4152-a5b9-2ee520b92b42","Type":"ContainerDied","Data":"7600d876981a90174fbeefc2146a03d5e3665d5803119a27ea5927d696d38043"}
Apr 16 16:16:35.188905 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.188884    2578 generic.go:358] "Generic (PLEG): container finished" podID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerID="d778e8385b8c7c288281e611fb6d3cbea071727b5b059eb1a1d9c3a43465a3de" exitCode=0
Apr 16 16:16:35.189021 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.188942    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb" event={"ID":"133b1df4-7046-4d19-8f72-860dc14f9d28","Type":"ContainerDied","Data":"d778e8385b8c7c288281e611fb6d3cbea071727b5b059eb1a1d9c3a43465a3de"}
Apr 16 16:16:35.190960 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.190936    2578 generic.go:358] "Generic (PLEG): container finished" podID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerID="193ab6b15131cfe1b28bdf6bb8a07b05fd2b62a5a69e0a50f2e45098244cd2d8" exitCode=0
Apr 16 16:16:35.191066 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.191046    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8" event={"ID":"74be648d-3de4-48fa-bcbb-19ce18467ca2","Type":"ContainerDied","Data":"193ab6b15131cfe1b28bdf6bb8a07b05fd2b62a5a69e0a50f2e45098244cd2d8"}
Apr 16 16:16:35.192902 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.192876    2578 generic.go:358] "Generic (PLEG): container finished" podID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerID="29b04b6d361b2c6380579ea66d12340a8f68382d413478a37ecf7eb3aaf4b1f0" exitCode=0
Apr 16 16:16:35.192991 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:35.192942    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f" event={"ID":"cbae3ae5-a40d-4bdb-bced-b644817a7fe5","Type":"ContainerDied","Data":"29b04b6d361b2c6380579ea66d12340a8f68382d413478a37ecf7eb3aaf4b1f0"}
Apr 16 16:16:36.198912 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.198875    2578 generic.go:358] "Generic (PLEG): container finished" podID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerID="0e6ee6b2246112816a21b5c6829bed927b3507af347e48a0a40451230bbeb073" exitCode=0
Apr 16 16:16:36.199398 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.198970    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb" event={"ID":"133b1df4-7046-4d19-8f72-860dc14f9d28","Type":"ContainerDied","Data":"0e6ee6b2246112816a21b5c6829bed927b3507af347e48a0a40451230bbeb073"}
Apr 16 16:16:36.342773 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.342735    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:36.385386 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.385364    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-util\") pod \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") "
Apr 16 16:16:36.385498 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.385427    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-bundle\") pod \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") "
Apr 16 16:16:36.385498 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.385481    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xd85\" (UniqueName: \"kubernetes.io/projected/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-kube-api-access-6xd85\") pod \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\" (UID: \"cbae3ae5-a40d-4bdb-bced-b644817a7fe5\") "
Apr 16 16:16:36.385959 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.385917    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-bundle" (OuterVolumeSpecName: "bundle") pod "cbae3ae5-a40d-4bdb-bced-b644817a7fe5" (UID: "cbae3ae5-a40d-4bdb-bced-b644817a7fe5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:36.387831 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.387808    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-kube-api-access-6xd85" (OuterVolumeSpecName: "kube-api-access-6xd85") pod "cbae3ae5-a40d-4bdb-bced-b644817a7fe5" (UID: "cbae3ae5-a40d-4bdb-bced-b644817a7fe5"). InnerVolumeSpecName "kube-api-access-6xd85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:16:36.388879 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.388860    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:36.391741 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.391714    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-util" (OuterVolumeSpecName: "util") pod "cbae3ae5-a40d-4bdb-bced-b644817a7fe5" (UID: "cbae3ae5-a40d-4bdb-bced-b644817a7fe5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:36.392431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.392414    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:36.486898 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.486874    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-bundle\") pod \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") "
Apr 16 16:16:36.487009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.486991    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-util\") pod \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") "
Apr 16 16:16:36.487054 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487040    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-util\") pod \"74be648d-3de4-48fa-bcbb-19ce18467ca2\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") "
Apr 16 16:16:36.487095 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487082    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-bundle\") pod \"74be648d-3de4-48fa-bcbb-19ce18467ca2\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") "
Apr 16 16:16:36.487150 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487107    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjr56\" (UniqueName: \"kubernetes.io/projected/74be648d-3de4-48fa-bcbb-19ce18467ca2-kube-api-access-vjr56\") pod \"74be648d-3de4-48fa-bcbb-19ce18467ca2\" (UID: \"74be648d-3de4-48fa-bcbb-19ce18467ca2\") "
Apr 16 16:16:36.487150 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487127    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr7tn\" (UniqueName: \"kubernetes.io/projected/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-kube-api-access-qr7tn\") pod \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\" (UID: \"54e5d39a-e3bd-4152-a5b9-2ee520b92b42\") "
Apr 16 16:16:36.487387 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487366    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.487494 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487392    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.487494 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487409    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xd85\" (UniqueName: \"kubernetes.io/projected/cbae3ae5-a40d-4bdb-bced-b644817a7fe5-kube-api-access-6xd85\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.487494 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487432    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-bundle" (OuterVolumeSpecName: "bundle") pod "54e5d39a-e3bd-4152-a5b9-2ee520b92b42" (UID: "54e5d39a-e3bd-4152-a5b9-2ee520b92b42"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:36.487828 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.487790    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-bundle" (OuterVolumeSpecName: "bundle") pod "74be648d-3de4-48fa-bcbb-19ce18467ca2" (UID: "74be648d-3de4-48fa-bcbb-19ce18467ca2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:36.489306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.489286    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74be648d-3de4-48fa-bcbb-19ce18467ca2-kube-api-access-vjr56" (OuterVolumeSpecName: "kube-api-access-vjr56") pod "74be648d-3de4-48fa-bcbb-19ce18467ca2" (UID: "74be648d-3de4-48fa-bcbb-19ce18467ca2"). InnerVolumeSpecName "kube-api-access-vjr56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:16:36.489680 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.489658    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-kube-api-access-qr7tn" (OuterVolumeSpecName: "kube-api-access-qr7tn") pod "54e5d39a-e3bd-4152-a5b9-2ee520b92b42" (UID: "54e5d39a-e3bd-4152-a5b9-2ee520b92b42"). InnerVolumeSpecName "kube-api-access-qr7tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:16:36.493120 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.493097    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-util" (OuterVolumeSpecName: "util") pod "74be648d-3de4-48fa-bcbb-19ce18467ca2" (UID: "74be648d-3de4-48fa-bcbb-19ce18467ca2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:36.493619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.493598    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-util" (OuterVolumeSpecName: "util") pod "54e5d39a-e3bd-4152-a5b9-2ee520b92b42" (UID: "54e5d39a-e3bd-4152-a5b9-2ee520b92b42"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:36.588618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.588596    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.588618 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.588615    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.588732 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.588624    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74be648d-3de4-48fa-bcbb-19ce18467ca2-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.588732 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.588632    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjr56\" (UniqueName: \"kubernetes.io/projected/74be648d-3de4-48fa-bcbb-19ce18467ca2-kube-api-access-vjr56\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.588732 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.588641    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7tn\" (UniqueName: \"kubernetes.io/projected/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-kube-api-access-qr7tn\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:36.588732 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:36.588650    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54e5d39a-e3bd-4152-a5b9-2ee520b92b42-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:37.203948 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.203912    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg" event={"ID":"54e5d39a-e3bd-4152-a5b9-2ee520b92b42","Type":"ContainerDied","Data":"65dd312969820259b3b155df962117449842cc45e2aa073bfa201fca6d1b0bbe"}
Apr 16 16:16:37.204276 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.203951    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65dd312969820259b3b155df962117449842cc45e2aa073bfa201fca6d1b0bbe"
Apr 16 16:16:37.204276 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.203949    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ef1189e9861fa30b9414ceb420c2d78e85403a7e10097f37afdacfec889cmmg"
Apr 16 16:16:37.205673 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.205652    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8"
Apr 16 16:16:37.205790 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.205668    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/309c32b5cc4cbd8148882f0ed2adbe6c47ee2761cc0c22c627755a3c30mlzx8" event={"ID":"74be648d-3de4-48fa-bcbb-19ce18467ca2","Type":"ContainerDied","Data":"f64a5f0099d34f4a782cfc7af20d079fb320cb52db623ea12d7398abfc3bc9b4"}
Apr 16 16:16:37.205790 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.205725    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f64a5f0099d34f4a782cfc7af20d079fb320cb52db623ea12d7398abfc3bc9b4"
Apr 16 16:16:37.207615 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.207591    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f" event={"ID":"cbae3ae5-a40d-4bdb-bced-b644817a7fe5","Type":"ContainerDied","Data":"632a1c8e68e500816874b26c42f116baa27362c9976c59b08e94721ad696cfe5"}
Apr 16 16:16:37.207615 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.207616    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632a1c8e68e500816874b26c42f116baa27362c9976c59b08e94721ad696cfe5"
Apr 16 16:16:37.207810 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.207786    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d54990151e6a3040f48962708f776cca7120e97625eb3a76d6fde2767bztw4f"
Apr 16 16:16:37.318758 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.318740    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:37.392811 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.392746    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-bundle\") pod \"133b1df4-7046-4d19-8f72-860dc14f9d28\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") "
Apr 16 16:16:37.392811 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.392786    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-util\") pod \"133b1df4-7046-4d19-8f72-860dc14f9d28\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") "
Apr 16 16:16:37.392953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.392819    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbtf4\" (UniqueName: \"kubernetes.io/projected/133b1df4-7046-4d19-8f72-860dc14f9d28-kube-api-access-lbtf4\") pod \"133b1df4-7046-4d19-8f72-860dc14f9d28\" (UID: \"133b1df4-7046-4d19-8f72-860dc14f9d28\") "
Apr 16 16:16:37.393405 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.393368    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-bundle" (OuterVolumeSpecName: "bundle") pod "133b1df4-7046-4d19-8f72-860dc14f9d28" (UID: "133b1df4-7046-4d19-8f72-860dc14f9d28"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:37.395028 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.394995    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133b1df4-7046-4d19-8f72-860dc14f9d28-kube-api-access-lbtf4" (OuterVolumeSpecName: "kube-api-access-lbtf4") pod "133b1df4-7046-4d19-8f72-860dc14f9d28" (UID: "133b1df4-7046-4d19-8f72-860dc14f9d28"). InnerVolumeSpecName "kube-api-access-lbtf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:16:37.397679 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.397638    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-util" (OuterVolumeSpecName: "util") pod "133b1df4-7046-4d19-8f72-860dc14f9d28" (UID: "133b1df4-7046-4d19-8f72-860dc14f9d28"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:16:37.494172 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.494147    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-bundle\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:37.494172 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.494167    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133b1df4-7046-4d19-8f72-860dc14f9d28-util\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:37.494291 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:37.494178    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtf4\" (UniqueName: \"kubernetes.io/projected/133b1df4-7046-4d19-8f72-860dc14f9d28-kube-api-access-lbtf4\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:16:38.213273 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:38.213234    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb" event={"ID":"133b1df4-7046-4d19-8f72-860dc14f9d28","Type":"ContainerDied","Data":"a7dbd725cb91152c7aa0fc834626a4a37afa94225d51f5c06928240f93c8c3b4"}
Apr 16 16:16:38.213273 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:38.213260    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/5b107722e7a95ad7441cca2226d3d656a8bf76dc65e15066bc01f8e5035tzzb"
Apr 16 16:16:38.213273 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:38.213273    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7dbd725cb91152c7aa0fc834626a4a37afa94225d51f5c06928240f93c8c3b4"
Apr 16 16:16:43.272174 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272146    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"]
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272458    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272470    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272479    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272484    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272492    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272498    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272510    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272515    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272522    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272527    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272535    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272539    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272546    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272551    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272557    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272562    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272569    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272573    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerName="util"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272579    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272584    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272590    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272595    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerName="pull"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272603    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272608    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272657    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="133b1df4-7046-4d19-8f72-860dc14f9d28" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272664    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="cbae3ae5-a40d-4bdb-bced-b644817a7fe5" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272671    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="54e5d39a-e3bd-4152-a5b9-2ee520b92b42" containerName="extract"
Apr 16 16:16:43.272788 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.272678    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="74be648d-3de4-48fa-bcbb-19ce18467ca2" containerName="extract"
Apr 16 16:16:43.277097 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.277080    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"
Apr 16 16:16:43.281268 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.281246    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Apr 16 16:16:43.282781 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.282762    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Apr 16 16:16:43.282943 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.282807    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bcrq9\""
Apr 16 16:16:43.289830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.289800    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"]
Apr 16 16:16:43.338634 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.338608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7d5h\" (UniqueName: \"kubernetes.io/projected/b9825944-320a-4eeb-90c2-72aa738b68d7-kube-api-access-c7d5h\") pod \"limitador-operator-controller-manager-c7fb4c8d5-88mwq\" (UID: \"b9825944-320a-4eeb-90c2-72aa738b68d7\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"
Apr 16 16:16:43.439375 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.439351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7d5h\" (UniqueName: \"kubernetes.io/projected/b9825944-320a-4eeb-90c2-72aa738b68d7-kube-api-access-c7d5h\") pod \"limitador-operator-controller-manager-c7fb4c8d5-88mwq\" (UID: \"b9825944-320a-4eeb-90c2-72aa738b68d7\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"
Apr 16 16:16:43.448667 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.448638    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7d5h\" (UniqueName: \"kubernetes.io/projected/b9825944-320a-4eeb-90c2-72aa738b68d7-kube-api-access-c7d5h\") pod \"limitador-operator-controller-manager-c7fb4c8d5-88mwq\" (UID: \"b9825944-320a-4eeb-90c2-72aa738b68d7\") " pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"
Apr 16 16:16:43.588064 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.588005    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"
Apr 16 16:16:43.718888 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:43.718860    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"]
Apr 16 16:16:43.720245 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:43.720219    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9825944_320a_4eeb_90c2_72aa738b68d7.slice/crio-97f53cf2c19d80634f709a76df22b580a2b842cca9f2ab8f7f40744c38a0fd23 WatchSource:0}: Error finding container 97f53cf2c19d80634f709a76df22b580a2b842cca9f2ab8f7f40744c38a0fd23: Status 404 returned error can't find the container with id 97f53cf2c19d80634f709a76df22b580a2b842cca9f2ab8f7f40744c38a0fd23
Apr 16 16:16:44.236410 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:44.236377    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq" event={"ID":"b9825944-320a-4eeb-90c2-72aa738b68d7","Type":"ContainerStarted","Data":"97f53cf2c19d80634f709a76df22b580a2b842cca9f2ab8f7f40744c38a0fd23"}
Apr 16 16:16:45.419257 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.419223    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"]
Apr 16 16:16:45.428306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.428284    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"
Apr 16 16:16:45.432180 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.432138    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n2t6g\""
Apr 16 16:16:45.432329 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.432308    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Apr 16 16:16:45.439122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.439097    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"]
Apr 16 16:16:45.556165 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.556126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzbqd\" (UniqueName: \"kubernetes.io/projected/d52e1c03-0b94-42f8-8233-43563211f01c-kube-api-access-jzbqd\") pod \"dns-operator-controller-manager-844548ff4c-vmcdg\" (UID: \"d52e1c03-0b94-42f8-8233-43563211f01c\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"
Apr 16 16:16:45.657461 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.657416    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jzbqd\" (UniqueName: \"kubernetes.io/projected/d52e1c03-0b94-42f8-8233-43563211f01c-kube-api-access-jzbqd\") pod \"dns-operator-controller-manager-844548ff4c-vmcdg\" (UID: \"d52e1c03-0b94-42f8-8233-43563211f01c\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"
Apr 16 16:16:45.671868 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.671813    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzbqd\" (UniqueName: \"kubernetes.io/projected/d52e1c03-0b94-42f8-8233-43563211f01c-kube-api-access-jzbqd\") pod \"dns-operator-controller-manager-844548ff4c-vmcdg\" (UID: \"d52e1c03-0b94-42f8-8233-43563211f01c\") " pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"
Apr 16 16:16:45.742683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:45.742651    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"
Apr 16 16:16:46.379588 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:46.379565    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"]
Apr 16 16:16:46.381319 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:46.381293    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd52e1c03_0b94_42f8_8233_43563211f01c.slice/crio-4c2908f27e2af8b71033df086f7f870cd97d520bb5349d7d0aa11afd5f8519e0 WatchSource:0}: Error finding container 4c2908f27e2af8b71033df086f7f870cd97d520bb5349d7d0aa11afd5f8519e0: Status 404 returned error can't find the container with id 4c2908f27e2af8b71033df086f7f870cd97d520bb5349d7d0aa11afd5f8519e0
Apr 16 16:16:47.249586 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:47.249551    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg" event={"ID":"d52e1c03-0b94-42f8-8233-43563211f01c","Type":"ContainerStarted","Data":"4c2908f27e2af8b71033df086f7f870cd97d520bb5349d7d0aa11afd5f8519e0"}
Apr 16 16:16:47.251306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:47.251272    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq" event={"ID":"b9825944-320a-4eeb-90c2-72aa738b68d7","Type":"ContainerStarted","Data":"652bd4e859565e202e1caeabb3e6087d04d5f1ccd0db19f1dceeca2b631ed0b7"}
Apr 16 16:16:47.251462 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:47.251391    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"
Apr 16 16:16:47.268921 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:47.268883    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq" podStartSLOduration=1.711179792 podStartE2EDuration="4.268869125s" podCreationTimestamp="2026-04-16 16:16:43 +0000 UTC" firstStartedPulling="2026-04-16 16:16:43.722302039 +0000 UTC m=+925.118386452" lastFinishedPulling="2026-04-16 16:16:46.279991378 +0000 UTC m=+927.676075785" observedRunningTime="2026-04-16 16:16:47.26776622 +0000 UTC m=+928.663850641" watchObservedRunningTime="2026-04-16 16:16:47.268869125 +0000 UTC m=+928.664953546"
Apr 16 16:16:48.256057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:48.256023    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg" event={"ID":"d52e1c03-0b94-42f8-8233-43563211f01c","Type":"ContainerStarted","Data":"1fa3eca458827b811674676e0d2cb790f6f244417da003c81949d16f65a6cf94"}
Apr 16 16:16:48.256404 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:48.256363    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"
Apr 16 16:16:48.274665 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:48.274615    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg" podStartSLOduration=1.908343672 podStartE2EDuration="3.274599067s" podCreationTimestamp="2026-04-16 16:16:45 +0000 UTC" firstStartedPulling="2026-04-16 16:16:46.383351472 +0000 UTC m=+927.779435870" lastFinishedPulling="2026-04-16 16:16:47.749606868 +0000 UTC m=+929.145691265" observedRunningTime="2026-04-16 16:16:48.274145099 +0000 UTC m=+929.670229520" watchObservedRunningTime="2026-04-16 16:16:48.274599067 +0000 UTC m=+929.670683488"
Apr 16 16:16:49.123220 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.123185    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"]
Apr 16 16:16:49.125734 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.125718    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.129142 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.129120    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-25j52\""
Apr 16 16:16:49.138667 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.138644    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"]
Apr 16 16:16:49.188534 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.188510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mft\" (UniqueName: \"kubernetes.io/projected/f4f3f8cd-567f-495b-ad16-031017f150f4-kube-api-access-x2mft\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-65q4c\" (UID: \"f4f3f8cd-567f-495b-ad16-031017f150f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.188662 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.188591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4f3f8cd-567f-495b-ad16-031017f150f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-65q4c\" (UID: \"f4f3f8cd-567f-495b-ad16-031017f150f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.289950 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.289924    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mft\" (UniqueName: \"kubernetes.io/projected/f4f3f8cd-567f-495b-ad16-031017f150f4-kube-api-access-x2mft\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-65q4c\" (UID: \"f4f3f8cd-567f-495b-ad16-031017f150f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.290287 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.289988    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4f3f8cd-567f-495b-ad16-031017f150f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-65q4c\" (UID: \"f4f3f8cd-567f-495b-ad16-031017f150f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.290377 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.290359    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f4f3f8cd-567f-495b-ad16-031017f150f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-65q4c\" (UID: \"f4f3f8cd-567f-495b-ad16-031017f150f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.298261 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.298231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mft\" (UniqueName: \"kubernetes.io/projected/f4f3f8cd-567f-495b-ad16-031017f150f4-kube-api-access-x2mft\") pod \"kuadrant-operator-controller-manager-6ddf9554fc-65q4c\" (UID: \"f4f3f8cd-567f-495b-ad16-031017f150f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.436765 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.436705    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:49.570196 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:49.570176    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"]
Apr 16 16:16:49.572468 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:16:49.572313    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f3f8cd_567f_495b_ad16_031017f150f4.slice/crio-388dec54716ecc1fa8e9c9dca5b871d0d9e252ec13b97eccde5462d5e9896516 WatchSource:0}: Error finding container 388dec54716ecc1fa8e9c9dca5b871d0d9e252ec13b97eccde5462d5e9896516: Status 404 returned error can't find the container with id 388dec54716ecc1fa8e9c9dca5b871d0d9e252ec13b97eccde5462d5e9896516
Apr 16 16:16:50.266171 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:50.266138    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c" event={"ID":"f4f3f8cd-567f-495b-ad16-031017f150f4","Type":"ContainerStarted","Data":"388dec54716ecc1fa8e9c9dca5b871d0d9e252ec13b97eccde5462d5e9896516"}
Apr 16 16:16:55.293414 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:55.293377    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c" event={"ID":"f4f3f8cd-567f-495b-ad16-031017f150f4","Type":"ContainerStarted","Data":"378ff8b41fbf8e81f4c3656a07da87bb7dab78ffc5ec3a38af2b7e7cbf9c7ec2"}
Apr 16 16:16:55.293767 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:55.293484    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:16:55.313486 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:55.313373    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c" podStartSLOduration=0.768830372 podStartE2EDuration="6.313359151s" podCreationTimestamp="2026-04-16 16:16:49 +0000 UTC" firstStartedPulling="2026-04-16 16:16:49.575458709 +0000 UTC m=+930.971543122" lastFinishedPulling="2026-04-16 16:16:55.119987502 +0000 UTC m=+936.516071901" observedRunningTime="2026-04-16 16:16:55.312404973 +0000 UTC m=+936.708489392" watchObservedRunningTime="2026-04-16 16:16:55.313359151 +0000 UTC m=+936.709443571"
Apr 16 16:16:58.258262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:16:58.258235    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-c7fb4c8d5-88mwq"
Apr 16 16:17:00.268709 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:00.268678    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-844548ff4c-vmcdg"
Apr 16 16:17:06.299311 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:06.299285    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6ddf9554fc-65q4c"
Apr 16 16:17:37.769430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.769383    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"]
Apr 16 16:17:37.773029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.773005    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:37.775562 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.775534    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Apr 16 16:17:37.775702 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.775534    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9p747\""
Apr 16 16:17:37.782250 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.782219    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"]
Apr 16 16:17:37.865960 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.865905    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"]
Apr 16 16:17:37.918920 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.918881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a238761-af2e-4007-9940-98d021ba8969-config-file\") pod \"limitador-limitador-64c8f475fb-7hh5p\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:37.918920 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:37.918927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpxq\" (UniqueName: \"kubernetes.io/projected/9a238761-af2e-4007-9940-98d021ba8969-kube-api-access-mfpxq\") pod \"limitador-limitador-64c8f475fb-7hh5p\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:38.019729 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:38.019625    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpxq\" (UniqueName: \"kubernetes.io/projected/9a238761-af2e-4007-9940-98d021ba8969-kube-api-access-mfpxq\") pod \"limitador-limitador-64c8f475fb-7hh5p\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:38.019887 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:38.019777    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a238761-af2e-4007-9940-98d021ba8969-config-file\") pod \"limitador-limitador-64c8f475fb-7hh5p\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:38.020541 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:38.020513    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a238761-af2e-4007-9940-98d021ba8969-config-file\") pod \"limitador-limitador-64c8f475fb-7hh5p\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:38.030268 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:38.030236    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpxq\" (UniqueName: \"kubernetes.io/projected/9a238761-af2e-4007-9940-98d021ba8969-kube-api-access-mfpxq\") pod \"limitador-limitador-64c8f475fb-7hh5p\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") " pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:38.086541 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:38.086504    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:38.221077 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:38.221041    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"]
Apr 16 16:17:38.222752 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:17:38.222721    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a238761_af2e_4007_9940_98d021ba8969.slice/crio-899d4bae990d1ef1b33cf00620c9ade58e2379212eac244ed1185b361263bfb6 WatchSource:0}: Error finding container 899d4bae990d1ef1b33cf00620c9ade58e2379212eac244ed1185b361263bfb6: Status 404 returned error can't find the container with id 899d4bae990d1ef1b33cf00620c9ade58e2379212eac244ed1185b361263bfb6
Apr 16 16:17:38.492887 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:38.492844    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p" event={"ID":"9a238761-af2e-4007-9940-98d021ba8969","Type":"ContainerStarted","Data":"899d4bae990d1ef1b33cf00620c9ade58e2379212eac244ed1185b361263bfb6"}
Apr 16 16:17:42.511343 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:42.511303    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p" event={"ID":"9a238761-af2e-4007-9940-98d021ba8969","Type":"ContainerStarted","Data":"4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc"}
Apr 16 16:17:42.511774 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:42.511430    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:42.527237 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:42.526852    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p" podStartSLOduration=1.76965583 podStartE2EDuration="5.526835891s" podCreationTimestamp="2026-04-16 16:17:37 +0000 UTC" firstStartedPulling="2026-04-16 16:17:38.225030448 +0000 UTC m=+979.621114853" lastFinishedPulling="2026-04-16 16:17:41.982210516 +0000 UTC m=+983.378294914" observedRunningTime="2026-04-16 16:17:42.526389349 +0000 UTC m=+983.922473770" watchObservedRunningTime="2026-04-16 16:17:42.526835891 +0000 UTC m=+983.922920312"
Apr 16 16:17:50.633671 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:50.633628    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"]
Apr 16 16:17:50.634155 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:50.633939    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p" podUID="9a238761-af2e-4007-9940-98d021ba8969" containerName="limitador" containerID="cri-o://4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc" gracePeriod=30
Apr 16 16:17:50.634737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:50.634617    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:51.181730 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.181704    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:51.336443 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.336414    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfpxq\" (UniqueName: \"kubernetes.io/projected/9a238761-af2e-4007-9940-98d021ba8969-kube-api-access-mfpxq\") pod \"9a238761-af2e-4007-9940-98d021ba8969\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") "
Apr 16 16:17:51.336644 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.336525    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a238761-af2e-4007-9940-98d021ba8969-config-file\") pod \"9a238761-af2e-4007-9940-98d021ba8969\" (UID: \"9a238761-af2e-4007-9940-98d021ba8969\") "
Apr 16 16:17:51.336892 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.336864    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a238761-af2e-4007-9940-98d021ba8969-config-file" (OuterVolumeSpecName: "config-file") pod "9a238761-af2e-4007-9940-98d021ba8969" (UID: "9a238761-af2e-4007-9940-98d021ba8969"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:17:51.338651 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.338628    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a238761-af2e-4007-9940-98d021ba8969-kube-api-access-mfpxq" (OuterVolumeSpecName: "kube-api-access-mfpxq") pod "9a238761-af2e-4007-9940-98d021ba8969" (UID: "9a238761-af2e-4007-9940-98d021ba8969"). InnerVolumeSpecName "kube-api-access-mfpxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:17:51.437814 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.437783    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfpxq\" (UniqueName: \"kubernetes.io/projected/9a238761-af2e-4007-9940-98d021ba8969-kube-api-access-mfpxq\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:17:51.437814 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.437815    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9a238761-af2e-4007-9940-98d021ba8969-config-file\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:17:51.549761 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.549728    2578 generic.go:358] "Generic (PLEG): container finished" podID="9a238761-af2e-4007-9940-98d021ba8969" containerID="4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc" exitCode=0
Apr 16 16:17:51.549898 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.549845    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"
Apr 16 16:17:51.549898 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.549861    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p" event={"ID":"9a238761-af2e-4007-9940-98d021ba8969","Type":"ContainerDied","Data":"4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc"}
Apr 16 16:17:51.550007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.549899    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c8f475fb-7hh5p" event={"ID":"9a238761-af2e-4007-9940-98d021ba8969","Type":"ContainerDied","Data":"899d4bae990d1ef1b33cf00620c9ade58e2379212eac244ed1185b361263bfb6"}
Apr 16 16:17:51.550007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.549921    2578 scope.go:117] "RemoveContainer" containerID="4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc"
Apr 16 16:17:51.559264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.559250    2578 scope.go:117] "RemoveContainer" containerID="4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc"
Apr 16 16:17:51.559548 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:17:51.559525    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc\": container with ID starting with 4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc not found: ID does not exist" containerID="4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc"
Apr 16 16:17:51.559641 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.559554    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc"} err="failed to get container status \"4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc\": rpc error: code = NotFound desc = could not find container \"4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc\": container with ID starting with 4d61bc98d4ceeb05d3b853d6d3ab1619da5daa8409bc7d7936808603c2af5abc not found: ID does not exist"
Apr 16 16:17:51.571744 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.571722    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"]
Apr 16 16:17:51.575105 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:51.575085    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c8f475fb-7hh5p"]
Apr 16 16:17:53.107352 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:17:53.107320    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a238761-af2e-4007-9940-98d021ba8969" path="/var/lib/kubelet/pods/9a238761-af2e-4007-9940-98d021ba8969/volumes"
Apr 16 16:18:09.452313 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.452279    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"]
Apr 16 16:18:09.452809 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.452651    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a238761-af2e-4007-9940-98d021ba8969" containerName="limitador"
Apr 16 16:18:09.452809 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.452662    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a238761-af2e-4007-9940-98d021ba8969" containerName="limitador"
Apr 16 16:18:09.452809 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.452716    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="9a238761-af2e-4007-9940-98d021ba8969" containerName="limitador"
Apr 16 16:18:09.459906 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.459877    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.470923 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.470897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.471047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.470940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.471047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.471010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzww\" (UniqueName: \"kubernetes.io/projected/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-kube-api-access-gpzww\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.471160 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.471061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.471160 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.471089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.471160 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.471125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.471160 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.471155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.474183 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.474159    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"]
Apr 16 16:18:09.572236 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.572207    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzww\" (UniqueName: \"kubernetes.io/projected/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-kube-api-access-gpzww\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.572366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.572242    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.572366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.572276    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.572366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.572311    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.572366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.572354    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.572591 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.572388    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.572591 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.572431    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.573208 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.573080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.574744 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.574714    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-local-certs\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.574898 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.574879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.574985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.574882    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-kubeconfig\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.575025 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.574978    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-cacerts\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.581756 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.581738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-istio-token\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.582376 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.582356    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzww\" (UniqueName: \"kubernetes.io/projected/8a19d4af-2b9e-4a5f-ae96-0567d9434a5f-kube-api-access-gpzww\") pod \"istiod-openshift-gateway-55ff986f96-8lbdh\" (UID: \"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f\") " pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.770753 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.770718    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:09.907812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.907782    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"]
Apr 16 16:18:09.908505 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:18:09.908476    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a19d4af_2b9e_4a5f_ae96_0567d9434a5f.slice/crio-763f8d6cbcb5ffc846f1a2b7c2bd1518ca81d5ad1735232a99348f4f2bf57086 WatchSource:0}: Error finding container 763f8d6cbcb5ffc846f1a2b7c2bd1518ca81d5ad1735232a99348f4f2bf57086: Status 404 returned error can't find the container with id 763f8d6cbcb5ffc846f1a2b7c2bd1518ca81d5ad1735232a99348f4f2bf57086
Apr 16 16:18:09.912433 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.911488    2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Apr 16 16:18:09.912433 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:09.911563    2578 kubelet_resources.go:45] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Apr 16 16:18:10.625015 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:10.624980    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh" event={"ID":"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f","Type":"ContainerStarted","Data":"5bc54fe9bd22e856a4e8ad068641306ca331e7770a88736fd9f6f644d4842c30"}
Apr 16 16:18:10.625015 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:10.625017    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh" event={"ID":"8a19d4af-2b9e-4a5f-ae96-0567d9434a5f","Type":"ContainerStarted","Data":"763f8d6cbcb5ffc846f1a2b7c2bd1518ca81d5ad1735232a99348f4f2bf57086"}
Apr 16 16:18:10.625537 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:10.625209    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:10.626940 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:10.626919    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-55ff986f96-8lbdh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Apr 16 16:18:10.627034 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:10.626961    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh" podUID="8a19d4af-2b9e-4a5f-ae96-0567d9434a5f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Apr 16 16:18:10.657318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:10.657277    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh" podStartSLOduration=1.6572649510000002 podStartE2EDuration="1.657264951s" podCreationTimestamp="2026-04-16 16:18:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:18:10.65095886 +0000 UTC m=+1012.047043280" watchObservedRunningTime="2026-04-16 16:18:10.657264951 +0000 UTC m=+1012.053349370"
Apr 16 16:18:11.629577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:11.629544    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-55ff986f96-8lbdh"
Apr 16 16:18:11.716911 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:11.716882    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"]
Apr 16 16:18:11.717132 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:11.717112    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l" podUID="584f000a-b973-44a3-900d-b8473287084c" containerName="discovery" containerID="cri-o://7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587" gracePeriod=30
Apr 16 16:18:11.986509 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:11.986486    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:18:12.095781 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.095752    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/584f000a-b973-44a3-900d-b8473287084c-local-certs\") pod \"584f000a-b973-44a3-900d-b8473287084c\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") "
Apr 16 16:18:12.096109 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.096083    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-kubeconfig\") pod \"584f000a-b973-44a3-900d-b8473287084c\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") "
Apr 16 16:18:12.096269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.096171    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/584f000a-b973-44a3-900d-b8473287084c-istio-csr-ca-configmap\") pod \"584f000a-b973-44a3-900d-b8473287084c\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") "
Apr 16 16:18:12.096269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.096237    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-istio-token\") pod \"584f000a-b973-44a3-900d-b8473287084c\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") "
Apr 16 16:18:12.096376 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.096287    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgbvq\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-kube-api-access-sgbvq\") pod \"584f000a-b973-44a3-900d-b8473287084c\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") "
Apr 16 16:18:12.096376 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.096327    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-cacerts\") pod \"584f000a-b973-44a3-900d-b8473287084c\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") "
Apr 16 16:18:12.096885 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.096390    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-csr-dns-cert\") pod \"584f000a-b973-44a3-900d-b8473287084c\" (UID: \"584f000a-b973-44a3-900d-b8473287084c\") "
Apr 16 16:18:12.097831 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.097410    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584f000a-b973-44a3-900d-b8473287084c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "584f000a-b973-44a3-900d-b8473287084c" (UID: "584f000a-b973-44a3-900d-b8473287084c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Apr 16 16:18:12.098152 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.098125    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/584f000a-b973-44a3-900d-b8473287084c-istio-csr-ca-configmap\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:12.099940 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.099888    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "584f000a-b973-44a3-900d-b8473287084c" (UID: "584f000a-b973-44a3-900d-b8473287084c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:18:12.105129 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.105034    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-cacerts" (OuterVolumeSpecName: "cacerts") pod "584f000a-b973-44a3-900d-b8473287084c" (UID: "584f000a-b973-44a3-900d-b8473287084c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:18:12.105129 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.105124    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/584f000a-b973-44a3-900d-b8473287084c-local-certs" (OuterVolumeSpecName: "local-certs") pod "584f000a-b973-44a3-900d-b8473287084c" (UID: "584f000a-b973-44a3-900d-b8473287084c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:18:12.105375 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.105304    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-istio-token" (OuterVolumeSpecName: "istio-token") pod "584f000a-b973-44a3-900d-b8473287084c" (UID: "584f000a-b973-44a3-900d-b8473287084c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:18:12.105590 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.105559    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-kube-api-access-sgbvq" (OuterVolumeSpecName: "kube-api-access-sgbvq") pod "584f000a-b973-44a3-900d-b8473287084c" (UID: "584f000a-b973-44a3-900d-b8473287084c"). InnerVolumeSpecName "kube-api-access-sgbvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:18:12.106582 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.106561    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "584f000a-b973-44a3-900d-b8473287084c" (UID: "584f000a-b973-44a3-900d-b8473287084c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:18:12.199353 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.199286    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-istio-token\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:12.199353 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.199314    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbvq\" (UniqueName: \"kubernetes.io/projected/584f000a-b973-44a3-900d-b8473287084c-kube-api-access-sgbvq\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:12.199353 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.199328    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-cacerts\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:12.199353 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.199343    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-csr-dns-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:12.199353 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.199356    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/584f000a-b973-44a3-900d-b8473287084c-local-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:12.199701 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.199371    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/584f000a-b973-44a3-900d-b8473287084c-istio-kubeconfig\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:12.634778 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.634738    2578 generic.go:358] "Generic (PLEG): container finished" podID="584f000a-b973-44a3-900d-b8473287084c" containerID="7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587" exitCode=0
Apr 16 16:18:12.635230 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.634821    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"
Apr 16 16:18:12.635230 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.634824    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l" event={"ID":"584f000a-b973-44a3-900d-b8473287084c","Type":"ContainerDied","Data":"7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587"}
Apr 16 16:18:12.635230 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.634873    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l" event={"ID":"584f000a-b973-44a3-900d-b8473287084c","Type":"ContainerDied","Data":"c2fdb73ed2abbb8ef0396db508076015d8a763486615e7a202da8b8b02edad4d"}
Apr 16 16:18:12.635230 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.634895    2578 scope.go:117] "RemoveContainer" containerID="7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587"
Apr 16 16:18:12.644504 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.644488    2578 scope.go:117] "RemoveContainer" containerID="7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587"
Apr 16 16:18:12.644748 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:18:12.644728    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587\": container with ID starting with 7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587 not found: ID does not exist" containerID="7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587"
Apr 16 16:18:12.644817 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.644756    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587"} err="failed to get container status \"7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587\": rpc error: code = NotFound desc = could not find container \"7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587\": container with ID starting with 7cd61eef9f16e4a753b068e49cef1962392863bd463bfcda4da8dc6025102587 not found: ID does not exist"
Apr 16 16:18:12.672056 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.672024    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"]
Apr 16 16:18:12.674209 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:12.674188    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7cd77c7ffd-pq77l"]
Apr 16 16:18:13.108387 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:13.108347    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584f000a-b973-44a3-900d-b8473287084c" path="/var/lib/kubelet/pods/584f000a-b973-44a3-900d-b8473287084c/volumes"
Apr 16 16:18:18.256706 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.256659    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-ms48w"]
Apr 16 16:18:18.257256 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.257233    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584f000a-b973-44a3-900d-b8473287084c" containerName="discovery"
Apr 16 16:18:18.257256 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.257257    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="584f000a-b973-44a3-900d-b8473287084c" containerName="discovery"
Apr 16 16:18:18.257408 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.257316    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="584f000a-b973-44a3-900d-b8473287084c" containerName="discovery"
Apr 16 16:18:18.271435 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.271417    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-ms48w"]
Apr 16 16:18:18.271564 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.271551    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.273919 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.273900    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Apr 16 16:18:18.274028 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.273927    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Apr 16 16:18:18.274028 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.273940    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-kjktr\""
Apr 16 16:18:18.274949 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.274932    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Apr 16 16:18:18.281121 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.281094    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-fc44f49f-m2vc9"]
Apr 16 16:18:18.286564 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.286436    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:18.289151 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.289130    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kj68m\""
Apr 16 16:18:18.289390 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.289375    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-fc44f49f-m2vc9"]
Apr 16 16:18:18.290279 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.290260    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Apr 16 16:18:18.323113 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.323093    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-86cc847c5c-5cmmr"]
Apr 16 16:18:18.326709 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.326692    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.329109 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.329086    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Apr 16 16:18:18.329192 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.329110    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bl9k5\""
Apr 16 16:18:18.335557 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.335535    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-5cmmr"]
Apr 16 16:18:18.347971 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.347950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e232330-014d-422b-b16e-1b250325c813-cert\") pod \"kserve-controller-manager-7c68cb4fc8-ms48w\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.348061 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.347986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97kh\" (UniqueName: \"kubernetes.io/projected/1e232330-014d-422b-b16e-1b250325c813-kube-api-access-p97kh\") pod \"kserve-controller-manager-7c68cb4fc8-ms48w\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.348102 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.348061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ng9n\" (UniqueName: \"kubernetes.io/projected/746f3884-ce3c-4397-b2d2-417c2c8782a5-kube-api-access-9ng9n\") pod \"llmisvc-controller-manager-fc44f49f-m2vc9\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") " pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:18.348102 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.348083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert\") pod \"llmisvc-controller-manager-fc44f49f-m2vc9\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") " pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:18.448708 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.448677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmlj\" (UniqueName: \"kubernetes.io/projected/83495ec6-95cf-4e61-bd9d-40ef9973c2d4-kube-api-access-vjmlj\") pod \"seaweedfs-86cc847c5c-5cmmr\" (UID: \"83495ec6-95cf-4e61-bd9d-40ef9973c2d4\") " pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.448830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.448724    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e232330-014d-422b-b16e-1b250325c813-cert\") pod \"kserve-controller-manager-7c68cb4fc8-ms48w\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.448830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.448771    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p97kh\" (UniqueName: \"kubernetes.io/projected/1e232330-014d-422b-b16e-1b250325c813-kube-api-access-p97kh\") pod \"kserve-controller-manager-7c68cb4fc8-ms48w\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.448830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.448790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83495ec6-95cf-4e61-bd9d-40ef9973c2d4-data\") pod \"seaweedfs-86cc847c5c-5cmmr\" (UID: \"83495ec6-95cf-4e61-bd9d-40ef9973c2d4\") " pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.448995 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.448846    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9ng9n\" (UniqueName: \"kubernetes.io/projected/746f3884-ce3c-4397-b2d2-417c2c8782a5-kube-api-access-9ng9n\") pod \"llmisvc-controller-manager-fc44f49f-m2vc9\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") " pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:18.448995 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.448875    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert\") pod \"llmisvc-controller-manager-fc44f49f-m2vc9\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") " pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:18.448995 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:18:18.448980    2578 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Apr 16 16:18:18.449158 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:18:18.449062    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert podName:746f3884-ce3c-4397-b2d2-417c2c8782a5 nodeName:}" failed. No retries permitted until 2026-04-16 16:18:18.949043305 +0000 UTC m=+1020.345127705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert") pod "llmisvc-controller-manager-fc44f49f-m2vc9" (UID: "746f3884-ce3c-4397-b2d2-417c2c8782a5") : secret "llmisvc-webhook-server-cert" not found
Apr 16 16:18:18.451401 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.451379    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e232330-014d-422b-b16e-1b250325c813-cert\") pod \"kserve-controller-manager-7c68cb4fc8-ms48w\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.458324 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.458291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ng9n\" (UniqueName: \"kubernetes.io/projected/746f3884-ce3c-4397-b2d2-417c2c8782a5-kube-api-access-9ng9n\") pod \"llmisvc-controller-manager-fc44f49f-m2vc9\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") " pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:18.458409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.458361    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97kh\" (UniqueName: \"kubernetes.io/projected/1e232330-014d-422b-b16e-1b250325c813-kube-api-access-p97kh\") pod \"kserve-controller-manager-7c68cb4fc8-ms48w\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.549621 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.549565    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmlj\" (UniqueName: \"kubernetes.io/projected/83495ec6-95cf-4e61-bd9d-40ef9973c2d4-kube-api-access-vjmlj\") pod \"seaweedfs-86cc847c5c-5cmmr\" (UID: \"83495ec6-95cf-4e61-bd9d-40ef9973c2d4\") " pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.549621 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.549615    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83495ec6-95cf-4e61-bd9d-40ef9973c2d4-data\") pod \"seaweedfs-86cc847c5c-5cmmr\" (UID: \"83495ec6-95cf-4e61-bd9d-40ef9973c2d4\") " pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.549914 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.549900    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/83495ec6-95cf-4e61-bd9d-40ef9973c2d4-data\") pod \"seaweedfs-86cc847c5c-5cmmr\" (UID: \"83495ec6-95cf-4e61-bd9d-40ef9973c2d4\") " pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.557308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.557281    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmlj\" (UniqueName: \"kubernetes.io/projected/83495ec6-95cf-4e61-bd9d-40ef9973c2d4-kube-api-access-vjmlj\") pod \"seaweedfs-86cc847c5c-5cmmr\" (UID: \"83495ec6-95cf-4e61-bd9d-40ef9973c2d4\") " pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.593270 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.593251    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:18.637269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.637242    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:18.727041 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.726992    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-ms48w"]
Apr 16 16:18:18.727977 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:18:18.727945    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e232330_014d_422b_b16e_1b250325c813.slice/crio-8be61474cf28bac4c24c34c0868ff9ae644db33e42711bbfeef0b182e3d58cd8 WatchSource:0}: Error finding container 8be61474cf28bac4c24c34c0868ff9ae644db33e42711bbfeef0b182e3d58cd8: Status 404 returned error can't find the container with id 8be61474cf28bac4c24c34c0868ff9ae644db33e42711bbfeef0b182e3d58cd8
Apr 16 16:18:18.771041 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.771021    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-86cc847c5c-5cmmr"]
Apr 16 16:18:18.774813 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:18:18.774783    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83495ec6_95cf_4e61_bd9d_40ef9973c2d4.slice/crio-0dc5a115e60136894e76975c99fbd2de2a6075ed64e237ff178b80ddec77c3fe WatchSource:0}: Error finding container 0dc5a115e60136894e76975c99fbd2de2a6075ed64e237ff178b80ddec77c3fe: Status 404 returned error can't find the container with id 0dc5a115e60136894e76975c99fbd2de2a6075ed64e237ff178b80ddec77c3fe
Apr 16 16:18:18.953187 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.953124    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert\") pod \"llmisvc-controller-manager-fc44f49f-m2vc9\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") " pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:18.955513 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:18.955492    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert\") pod \"llmisvc-controller-manager-fc44f49f-m2vc9\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") " pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:19.201575 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:19.201552    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kj68m\""
Apr 16 16:18:19.209668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:19.209646    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:19.503753 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:19.503722    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-fc44f49f-m2vc9"]
Apr 16 16:18:19.509869 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:18:19.509835    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod746f3884_ce3c_4397_b2d2_417c2c8782a5.slice/crio-3305d726a1053f9cfea982cc33ea70655c02bc52d8123e3923bcb31e8c4b2d16 WatchSource:0}: Error finding container 3305d726a1053f9cfea982cc33ea70655c02bc52d8123e3923bcb31e8c4b2d16: Status 404 returned error can't find the container with id 3305d726a1053f9cfea982cc33ea70655c02bc52d8123e3923bcb31e8c4b2d16
Apr 16 16:18:19.668057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:19.668023    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9" event={"ID":"746f3884-ce3c-4397-b2d2-417c2c8782a5","Type":"ContainerStarted","Data":"3305d726a1053f9cfea982cc33ea70655c02bc52d8123e3923bcb31e8c4b2d16"}
Apr 16 16:18:19.669585 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:19.669559    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-5cmmr" event={"ID":"83495ec6-95cf-4e61-bd9d-40ef9973c2d4","Type":"ContainerStarted","Data":"0dc5a115e60136894e76975c99fbd2de2a6075ed64e237ff178b80ddec77c3fe"}
Apr 16 16:18:19.671608 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:19.671567    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w" event={"ID":"1e232330-014d-422b-b16e-1b250325c813","Type":"ContainerStarted","Data":"8be61474cf28bac4c24c34c0868ff9ae644db33e42711bbfeef0b182e3d58cd8"}
Apr 16 16:18:23.001833 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:23.001798    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Apr 16 16:18:23.691321 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:23.691224    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-86cc847c5c-5cmmr" event={"ID":"83495ec6-95cf-4e61-bd9d-40ef9973c2d4","Type":"ContainerStarted","Data":"6dc35f63c33d94f0313c4baf0df89e2498ae2d0f49dc06aa7552ea50f7b0b036"}
Apr 16 16:18:23.691321 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:23.691291    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:23.692607 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:23.692578    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w" event={"ID":"1e232330-014d-422b-b16e-1b250325c813","Type":"ContainerStarted","Data":"b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799"}
Apr 16 16:18:23.692745 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:23.692697    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:23.709153 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:23.709102    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-86cc847c5c-5cmmr" podStartSLOduration=1.486222675 podStartE2EDuration="5.709086848s" podCreationTimestamp="2026-04-16 16:18:18 +0000 UTC" firstStartedPulling="2026-04-16 16:18:18.776362014 +0000 UTC m=+1020.172446412" lastFinishedPulling="2026-04-16 16:18:22.999225983 +0000 UTC m=+1024.395310585" observedRunningTime="2026-04-16 16:18:23.705958426 +0000 UTC m=+1025.102042848" watchObservedRunningTime="2026-04-16 16:18:23.709086848 +0000 UTC m=+1025.105171270"
Apr 16 16:18:23.722367 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:23.722320    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w" podStartSLOduration=1.871245582 podStartE2EDuration="5.722309454s" podCreationTimestamp="2026-04-16 16:18:18 +0000 UTC" firstStartedPulling="2026-04-16 16:18:18.729345973 +0000 UTC m=+1020.125430374" lastFinishedPulling="2026-04-16 16:18:22.580409831 +0000 UTC m=+1023.976494246" observedRunningTime="2026-04-16 16:18:23.719903888 +0000 UTC m=+1025.115988311" watchObservedRunningTime="2026-04-16 16:18:23.722309454 +0000 UTC m=+1025.118393875"
Apr 16 16:18:25.703187 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:25.703104    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9" event={"ID":"746f3884-ce3c-4397-b2d2-417c2c8782a5","Type":"ContainerStarted","Data":"1f1147d2f00ac5fcdb8c4ed565aecb625693195e2e44dddb56b2ab33908fe7ee"}
Apr 16 16:18:25.703569 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:25.703235    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:25.721145 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:25.721103    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9" podStartSLOduration=1.909255598 podStartE2EDuration="7.721089056s" podCreationTimestamp="2026-04-16 16:18:18 +0000 UTC" firstStartedPulling="2026-04-16 16:18:19.512058785 +0000 UTC m=+1020.908143191" lastFinishedPulling="2026-04-16 16:18:25.323892249 +0000 UTC m=+1026.719976649" observedRunningTime="2026-04-16 16:18:25.719738683 +0000 UTC m=+1027.115823104" watchObservedRunningTime="2026-04-16 16:18:25.721089056 +0000 UTC m=+1027.117173475"
Apr 16 16:18:29.700116 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:29.700087    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-86cc847c5c-5cmmr"
Apr 16 16:18:54.704011 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:54.703979    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:56.711513 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:56.711480    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:18:58.203664 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.203628    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-ms48w"]
Apr 16 16:18:58.204077 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.203855    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w" podUID="1e232330-014d-422b-b16e-1b250325c813" containerName="manager" containerID="cri-o://b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799" gracePeriod=10
Apr 16 16:18:58.227154 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.227124    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-lvg26"]
Apr 16 16:18:58.230608 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.230592    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.239826 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.239799    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-lvg26"]
Apr 16 16:18:58.372573 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.372526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxf8\" (UniqueName: \"kubernetes.io/projected/6523b077-a1cc-4863-9be5-4ffacd4a2d0e-kube-api-access-6jxf8\") pod \"kserve-controller-manager-7c68cb4fc8-lvg26\" (UID: \"6523b077-a1cc-4863-9be5-4ffacd4a2d0e\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.372731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.372625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6523b077-a1cc-4863-9be5-4ffacd4a2d0e-cert\") pod \"kserve-controller-manager-7c68cb4fc8-lvg26\" (UID: \"6523b077-a1cc-4863-9be5-4ffacd4a2d0e\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.452514 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.452488    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:58.473830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.473801    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6523b077-a1cc-4863-9be5-4ffacd4a2d0e-cert\") pod \"kserve-controller-manager-7c68cb4fc8-lvg26\" (UID: \"6523b077-a1cc-4863-9be5-4ffacd4a2d0e\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.473970 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.473894    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxf8\" (UniqueName: \"kubernetes.io/projected/6523b077-a1cc-4863-9be5-4ffacd4a2d0e-kube-api-access-6jxf8\") pod \"kserve-controller-manager-7c68cb4fc8-lvg26\" (UID: \"6523b077-a1cc-4863-9be5-4ffacd4a2d0e\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.476763 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.476727    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6523b077-a1cc-4863-9be5-4ffacd4a2d0e-cert\") pod \"kserve-controller-manager-7c68cb4fc8-lvg26\" (UID: \"6523b077-a1cc-4863-9be5-4ffacd4a2d0e\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.482189 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.482160    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxf8\" (UniqueName: \"kubernetes.io/projected/6523b077-a1cc-4863-9be5-4ffacd4a2d0e-kube-api-access-6jxf8\") pod \"kserve-controller-manager-7c68cb4fc8-lvg26\" (UID: \"6523b077-a1cc-4863-9be5-4ffacd4a2d0e\") " pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.575220 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.575172    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e232330-014d-422b-b16e-1b250325c813-cert\") pod \"1e232330-014d-422b-b16e-1b250325c813\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") "
Apr 16 16:18:58.575404 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.575281    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p97kh\" (UniqueName: \"kubernetes.io/projected/1e232330-014d-422b-b16e-1b250325c813-kube-api-access-p97kh\") pod \"1e232330-014d-422b-b16e-1b250325c813\" (UID: \"1e232330-014d-422b-b16e-1b250325c813\") "
Apr 16 16:18:58.577753 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.577723    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e232330-014d-422b-b16e-1b250325c813-kube-api-access-p97kh" (OuterVolumeSpecName: "kube-api-access-p97kh") pod "1e232330-014d-422b-b16e-1b250325c813" (UID: "1e232330-014d-422b-b16e-1b250325c813"). InnerVolumeSpecName "kube-api-access-p97kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:18:58.577753 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.577732    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e232330-014d-422b-b16e-1b250325c813-cert" (OuterVolumeSpecName: "cert") pod "1e232330-014d-422b-b16e-1b250325c813" (UID: "1e232330-014d-422b-b16e-1b250325c813"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:18:58.598025 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.598003    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:58.677235 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.677195    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97kh\" (UniqueName: \"kubernetes.io/projected/1e232330-014d-422b-b16e-1b250325c813-kube-api-access-p97kh\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:58.677385 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.677243    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e232330-014d-422b-b16e-1b250325c813-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:18:58.720946 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.720923    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-lvg26"]
Apr 16 16:18:58.722285 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:18:58.722255    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6523b077_a1cc_4863_9be5_4ffacd4a2d0e.slice/crio-97a76f3df4a72ebb0a97e911245e11ef08cfed39fdae96d4ef2f9e03e4cf09e9 WatchSource:0}: Error finding container 97a76f3df4a72ebb0a97e911245e11ef08cfed39fdae96d4ef2f9e03e4cf09e9: Status 404 returned error can't find the container with id 97a76f3df4a72ebb0a97e911245e11ef08cfed39fdae96d4ef2f9e03e4cf09e9
Apr 16 16:18:58.723600 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.723583    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:18:58.835041 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.835006    2578 generic.go:358] "Generic (PLEG): container finished" podID="1e232330-014d-422b-b16e-1b250325c813" containerID="b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799" exitCode=0
Apr 16 16:18:58.835223 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.835080    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w"
Apr 16 16:18:58.835223 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.835099    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w" event={"ID":"1e232330-014d-422b-b16e-1b250325c813","Type":"ContainerDied","Data":"b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799"}
Apr 16 16:18:58.835223 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.835135    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c68cb4fc8-ms48w" event={"ID":"1e232330-014d-422b-b16e-1b250325c813","Type":"ContainerDied","Data":"8be61474cf28bac4c24c34c0868ff9ae644db33e42711bbfeef0b182e3d58cd8"}
Apr 16 16:18:58.835223 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.835158    2578 scope.go:117] "RemoveContainer" containerID="b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799"
Apr 16 16:18:58.836349 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.836324    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26" event={"ID":"6523b077-a1cc-4863-9be5-4ffacd4a2d0e","Type":"ContainerStarted","Data":"97a76f3df4a72ebb0a97e911245e11ef08cfed39fdae96d4ef2f9e03e4cf09e9"}
Apr 16 16:18:58.844549 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.844532    2578 scope.go:117] "RemoveContainer" containerID="b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799"
Apr 16 16:18:58.844813 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:18:58.844793    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799\": container with ID starting with b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799 not found: ID does not exist" containerID="b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799"
Apr 16 16:18:58.844855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.844821    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799"} err="failed to get container status \"b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799\": rpc error: code = NotFound desc = could not find container \"b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799\": container with ID starting with b9392a4aed82472ed9150004570ec22b02dcaa398b9f272d02218e6398bfa799 not found: ID does not exist"
Apr 16 16:18:58.857505 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.857478    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-ms48w"]
Apr 16 16:18:58.860711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:58.860689    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7c68cb4fc8-ms48w"]
Apr 16 16:18:59.108724 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:59.108645    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e232330-014d-422b-b16e-1b250325c813" path="/var/lib/kubelet/pods/1e232330-014d-422b-b16e-1b250325c813/volumes"
Apr 16 16:18:59.841178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:59.841139    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26" event={"ID":"6523b077-a1cc-4863-9be5-4ffacd4a2d0e","Type":"ContainerStarted","Data":"307c394cb21af1a10bb3d02b78e0cec6111e03f14f1e1744bdcb115d6ae0968a"}
Apr 16 16:18:59.841178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:59.841193    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:18:59.857950 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:18:59.857906    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26" podStartSLOduration=1.405241781 podStartE2EDuration="1.85789138s" podCreationTimestamp="2026-04-16 16:18:58 +0000 UTC" firstStartedPulling="2026-04-16 16:18:58.723725468 +0000 UTC m=+1060.119809866" lastFinishedPulling="2026-04-16 16:18:59.176375067 +0000 UTC m=+1060.572459465" observedRunningTime="2026-04-16 16:18:59.855941654 +0000 UTC m=+1061.252026074" watchObservedRunningTime="2026-04-16 16:18:59.85789138 +0000 UTC m=+1061.253975799"
Apr 16 16:19:30.852030 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:30.851994    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7c68cb4fc8-lvg26"
Apr 16 16:19:31.747305 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.747274    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-d6n4q"]
Apr 16 16:19:31.747708 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.747693    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e232330-014d-422b-b16e-1b250325c813" containerName="manager"
Apr 16 16:19:31.747763 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.747710    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e232330-014d-422b-b16e-1b250325c813" containerName="manager"
Apr 16 16:19:31.747799 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.747780    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="1e232330-014d-422b-b16e-1b250325c813" containerName="manager"
Apr 16 16:19:31.751053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.751031    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:31.753656 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.753634    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-s4b5j\""
Apr 16 16:19:31.753759 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.753665    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Apr 16 16:19:31.759047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.758532    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-d6n4q"]
Apr 16 16:19:31.761460 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.761419    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-5rtdj"]
Apr 16 16:19:31.764651 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.764633    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:31.767101 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.767080    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-dgbdq\""
Apr 16 16:19:31.767212 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.767116    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Apr 16 16:19:31.773777 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.773746    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-5rtdj"]
Apr 16 16:19:31.858117 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.858076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwtv\" (UniqueName: \"kubernetes.io/projected/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-kube-api-access-ftwtv\") pod \"odh-model-controller-696fc77849-5rtdj\" (UID: \"a8121fdb-6482-4d89-8b1c-25d3e39b06d8\") " pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:31.858540 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.858131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4dd\" (UniqueName: \"kubernetes.io/projected/77254d9f-3abe-49b0-ac5c-a0385b611ddf-kube-api-access-dp4dd\") pod \"model-serving-api-86f7b4b499-d6n4q\" (UID: \"77254d9f-3abe-49b0-ac5c-a0385b611ddf\") " pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:31.858540 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.858240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-cert\") pod \"odh-model-controller-696fc77849-5rtdj\" (UID: \"a8121fdb-6482-4d89-8b1c-25d3e39b06d8\") " pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:31.858540 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.858280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77254d9f-3abe-49b0-ac5c-a0385b611ddf-tls-certs\") pod \"model-serving-api-86f7b4b499-d6n4q\" (UID: \"77254d9f-3abe-49b0-ac5c-a0385b611ddf\") " pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:31.959278 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.959231    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4dd\" (UniqueName: \"kubernetes.io/projected/77254d9f-3abe-49b0-ac5c-a0385b611ddf-kube-api-access-dp4dd\") pod \"model-serving-api-86f7b4b499-d6n4q\" (UID: \"77254d9f-3abe-49b0-ac5c-a0385b611ddf\") " pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:31.959485 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.959309    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-cert\") pod \"odh-model-controller-696fc77849-5rtdj\" (UID: \"a8121fdb-6482-4d89-8b1c-25d3e39b06d8\") " pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:31.959485 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.959334    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77254d9f-3abe-49b0-ac5c-a0385b611ddf-tls-certs\") pod \"model-serving-api-86f7b4b499-d6n4q\" (UID: \"77254d9f-3abe-49b0-ac5c-a0385b611ddf\") " pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:31.959485 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.959369    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwtv\" (UniqueName: \"kubernetes.io/projected/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-kube-api-access-ftwtv\") pod \"odh-model-controller-696fc77849-5rtdj\" (UID: \"a8121fdb-6482-4d89-8b1c-25d3e39b06d8\") " pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:31.959619 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:19:31.959491    2578 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Apr 16 16:19:31.959619 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:19:31.959491    2578 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Apr 16 16:19:31.959619 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:19:31.959584    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77254d9f-3abe-49b0-ac5c-a0385b611ddf-tls-certs podName:77254d9f-3abe-49b0-ac5c-a0385b611ddf nodeName:}" failed. No retries permitted until 2026-04-16 16:19:32.459547115 +0000 UTC m=+1093.855631513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/77254d9f-3abe-49b0-ac5c-a0385b611ddf-tls-certs") pod "model-serving-api-86f7b4b499-d6n4q" (UID: "77254d9f-3abe-49b0-ac5c-a0385b611ddf") : secret "model-serving-api-tls" not found
Apr 16 16:19:31.959752 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:19:31.959621    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-cert podName:a8121fdb-6482-4d89-8b1c-25d3e39b06d8 nodeName:}" failed. No retries permitted until 2026-04-16 16:19:32.45960638 +0000 UTC m=+1093.855690779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-cert") pod "odh-model-controller-696fc77849-5rtdj" (UID: "a8121fdb-6482-4d89-8b1c-25d3e39b06d8") : secret "odh-model-controller-webhook-cert" not found
Apr 16 16:19:31.969418 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.969390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4dd\" (UniqueName: \"kubernetes.io/projected/77254d9f-3abe-49b0-ac5c-a0385b611ddf-kube-api-access-dp4dd\") pod \"model-serving-api-86f7b4b499-d6n4q\" (UID: \"77254d9f-3abe-49b0-ac5c-a0385b611ddf\") " pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:31.969960 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:31.969943    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwtv\" (UniqueName: \"kubernetes.io/projected/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-kube-api-access-ftwtv\") pod \"odh-model-controller-696fc77849-5rtdj\" (UID: \"a8121fdb-6482-4d89-8b1c-25d3e39b06d8\") " pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:32.464239 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.464205    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-cert\") pod \"odh-model-controller-696fc77849-5rtdj\" (UID: \"a8121fdb-6482-4d89-8b1c-25d3e39b06d8\") " pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:32.464420 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.464246    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77254d9f-3abe-49b0-ac5c-a0385b611ddf-tls-certs\") pod \"model-serving-api-86f7b4b499-d6n4q\" (UID: \"77254d9f-3abe-49b0-ac5c-a0385b611ddf\") " pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:32.466757 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.466738    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8121fdb-6482-4d89-8b1c-25d3e39b06d8-cert\") pod \"odh-model-controller-696fc77849-5rtdj\" (UID: \"a8121fdb-6482-4d89-8b1c-25d3e39b06d8\") " pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:32.466884 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.466867    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77254d9f-3abe-49b0-ac5c-a0385b611ddf-tls-certs\") pod \"model-serving-api-86f7b4b499-d6n4q\" (UID: \"77254d9f-3abe-49b0-ac5c-a0385b611ddf\") " pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:32.663801 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.663765    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:32.676266 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.676222    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:32.808335 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.808304    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-d6n4q"]
Apr 16 16:19:32.811495 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:19:32.811462    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77254d9f_3abe_49b0_ac5c_a0385b611ddf.slice/crio-ad0d5f5296d9f608c67b71da363c84c0c30a4254b40758deeecbcae87fdbb6f6 WatchSource:0}: Error finding container ad0d5f5296d9f608c67b71da363c84c0c30a4254b40758deeecbcae87fdbb6f6: Status 404 returned error can't find the container with id ad0d5f5296d9f608c67b71da363c84c0c30a4254b40758deeecbcae87fdbb6f6
Apr 16 16:19:32.826007 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.825984    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-5rtdj"]
Apr 16 16:19:32.827069 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:19:32.827045    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8121fdb_6482_4d89_8b1c_25d3e39b06d8.slice/crio-2d14d7e2737fc9c558dafb5d3d2f134c10efb48940635745ea99d21e3a7eca08 WatchSource:0}: Error finding container 2d14d7e2737fc9c558dafb5d3d2f134c10efb48940635745ea99d21e3a7eca08: Status 404 returned error can't find the container with id 2d14d7e2737fc9c558dafb5d3d2f134c10efb48940635745ea99d21e3a7eca08
Apr 16 16:19:32.974145 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.974048    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-5rtdj" event={"ID":"a8121fdb-6482-4d89-8b1c-25d3e39b06d8","Type":"ContainerStarted","Data":"2d14d7e2737fc9c558dafb5d3d2f134c10efb48940635745ea99d21e3a7eca08"}
Apr 16 16:19:32.975291 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:32.975257    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-d6n4q" event={"ID":"77254d9f-3abe-49b0-ac5c-a0385b611ddf","Type":"ContainerStarted","Data":"ad0d5f5296d9f608c67b71da363c84c0c30a4254b40758deeecbcae87fdbb6f6"}
Apr 16 16:19:36.999530 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:36.999482    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-d6n4q" event={"ID":"77254d9f-3abe-49b0-ac5c-a0385b611ddf","Type":"ContainerStarted","Data":"659cd8a4825587f9092d6c5ecbe218f9e434208c3584ccf576bf3b610aab567c"}
Apr 16 16:19:37.000040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:36.999611    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:19:37.001019 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:37.000995    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-5rtdj" event={"ID":"a8121fdb-6482-4d89-8b1c-25d3e39b06d8","Type":"ContainerStarted","Data":"3a4388096390b5cb4767a82bfd1ace6518adca3e6bcc642246af4375b50df3e7"}
Apr 16 16:19:37.001175 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:37.001155    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:37.018421 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:37.018370    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-d6n4q" podStartSLOduration=2.237876637 podStartE2EDuration="6.01835637s" podCreationTimestamp="2026-04-16 16:19:31 +0000 UTC" firstStartedPulling="2026-04-16 16:19:32.813407706 +0000 UTC m=+1094.209492105" lastFinishedPulling="2026-04-16 16:19:36.593887437 +0000 UTC m=+1097.989971838" observedRunningTime="2026-04-16 16:19:37.015233719 +0000 UTC m=+1098.411318141" watchObservedRunningTime="2026-04-16 16:19:37.01835637 +0000 UTC m=+1098.414440789"
Apr 16 16:19:37.031026 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:37.030957    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-5rtdj" podStartSLOduration=2.2607229589999998 podStartE2EDuration="6.030941493s" podCreationTimestamp="2026-04-16 16:19:31 +0000 UTC" firstStartedPulling="2026-04-16 16:19:32.828338024 +0000 UTC m=+1094.224422423" lastFinishedPulling="2026-04-16 16:19:36.598556559 +0000 UTC m=+1097.994640957" observedRunningTime="2026-04-16 16:19:37.030485958 +0000 UTC m=+1098.426570381" watchObservedRunningTime="2026-04-16 16:19:37.030941493 +0000 UTC m=+1098.427025909"
Apr 16 16:19:48.007082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:48.007052    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-5rtdj"
Apr 16 16:19:48.008837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:19:48.008813    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-d6n4q"
Apr 16 16:20:21.318583 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.318547    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"]
Apr 16 16:20:21.322852 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.322824    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.325952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.325724    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-xfkst\""
Apr 16 16:20:21.325952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.325737    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 16 16:20:21.325952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.325792    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Apr 16 16:20:21.325952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.325735    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 16 16:20:21.325952 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.325737    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2fdlg\""
Apr 16 16:20:21.334634 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.334606    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"]
Apr 16 16:20:21.405675 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.405638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.405872 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.405717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.405872 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.405746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zn94\" (UniqueName: \"kubernetes.io/projected/e76779fa-5562-44e6-abbd-9fb7b54768cc-kube-api-access-5zn94\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.405872 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.405811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76779fa-5562-44e6-abbd-9fb7b54768cc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.405872 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.405864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.406012 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.405890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.506833 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.506797    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.506833 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.506837    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5zn94\" (UniqueName: \"kubernetes.io/projected/e76779fa-5562-44e6-abbd-9fb7b54768cc-kube-api-access-5zn94\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.506862    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76779fa-5562-44e6-abbd-9fb7b54768cc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.506989    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.507051    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507241 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.507123    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507241 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.507222    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507391 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.507372    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507487 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.507423    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-tmp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.507536 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.507424    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-cache\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.509645 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.509615    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76779fa-5562-44e6-abbd-9fb7b54768cc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.516777 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.516751    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zn94\" (UniqueName: \"kubernetes.io/projected/e76779fa-5562-44e6-abbd-9fb7b54768cc-kube-api-access-5zn94\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.637318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.637212    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:20:21.787364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:21.787333    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"]
Apr 16 16:20:21.789033 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:20:21.789005    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76779fa_5562_44e6_abbd_9fb7b54768cc.slice/crio-e979e6c79911304968a644cdd7062843914e5b6c4f5e062d1d242f10cf647221 WatchSource:0}: Error finding container e979e6c79911304968a644cdd7062843914e5b6c4f5e062d1d242f10cf647221: Status 404 returned error can't find the container with id e979e6c79911304968a644cdd7062843914e5b6c4f5e062d1d242f10cf647221
Apr 16 16:20:22.200630 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:22.200581    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerStarted","Data":"e979e6c79911304968a644cdd7062843914e5b6c4f5e062d1d242f10cf647221"}
Apr 16 16:20:26.225025 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:26.224975    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerStarted","Data":"722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6"}
Apr 16 16:20:27.230842 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:27.230805    2578 generic.go:358] "Generic (PLEG): container finished" podID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerID="722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6" exitCode=0
Apr 16 16:20:27.231260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:27.230892    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerDied","Data":"722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6"}
Apr 16 16:20:29.244792 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:20:29.244724    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerStarted","Data":"98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad"}
Apr 16 16:21:00.397727 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:00.397623    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerStarted","Data":"a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930"}
Apr 16 16:21:00.398214 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:00.397848    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:00.400832 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:00.400810    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:00.417811 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:00.417756    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" podStartSLOduration=1.200459176 podStartE2EDuration="39.417739563s" podCreationTimestamp="2026-04-16 16:20:21 +0000 UTC" firstStartedPulling="2026-04-16 16:20:21.791519383 +0000 UTC m=+1143.187603781" lastFinishedPulling="2026-04-16 16:21:00.008799752 +0000 UTC m=+1181.404884168" observedRunningTime="2026-04-16 16:21:00.416159842 +0000 UTC m=+1181.812244268" watchObservedRunningTime="2026-04-16 16:21:00.417739563 +0000 UTC m=+1181.813824075"
Apr 16 16:21:01.637737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:01.637701    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:01.637737 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:01.637746    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:11.639117 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:11.639086    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:11.640403 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:11.640375    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:12.586592 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:12.586555    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"]
Apr 16 16:21:13.450610 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:13.450556    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="main" containerID="cri-o://98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad" gracePeriod=30
Apr 16 16:21:13.451081 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:13.450584    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="tokenizer" containerID="cri-o://a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930" gracePeriod=30
Apr 16 16:21:14.456544 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.456514    2578 generic.go:358] "Generic (PLEG): container finished" podID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerID="98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad" exitCode=0
Apr 16 16:21:14.456856 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.456583    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerDied","Data":"98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad"}
Apr 16 16:21:14.594685 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.594663    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:14.731159 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731131    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-cache\") pod \"e76779fa-5562-44e6-abbd-9fb7b54768cc\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") "
Apr 16 16:21:14.731306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731170    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76779fa-5562-44e6-abbd-9fb7b54768cc-tls-certs\") pod \"e76779fa-5562-44e6-abbd-9fb7b54768cc\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") "
Apr 16 16:21:14.731306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731220    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-tmp\") pod \"e76779fa-5562-44e6-abbd-9fb7b54768cc\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") "
Apr 16 16:21:14.731306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731251    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-uds\") pod \"e76779fa-5562-44e6-abbd-9fb7b54768cc\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") "
Apr 16 16:21:14.731306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731291    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-kserve-provision-location\") pod \"e76779fa-5562-44e6-abbd-9fb7b54768cc\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") "
Apr 16 16:21:14.731567 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731322    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zn94\" (UniqueName: \"kubernetes.io/projected/e76779fa-5562-44e6-abbd-9fb7b54768cc-kube-api-access-5zn94\") pod \"e76779fa-5562-44e6-abbd-9fb7b54768cc\" (UID: \"e76779fa-5562-44e6-abbd-9fb7b54768cc\") "
Apr 16 16:21:14.731567 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731391    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "e76779fa-5562-44e6-abbd-9fb7b54768cc" (UID: "e76779fa-5562-44e6-abbd-9fb7b54768cc"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:14.731567 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731518    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:14.731728 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731562    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e76779fa-5562-44e6-abbd-9fb7b54768cc" (UID: "e76779fa-5562-44e6-abbd-9fb7b54768cc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:14.731728 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.731644    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "e76779fa-5562-44e6-abbd-9fb7b54768cc" (UID: "e76779fa-5562-44e6-abbd-9fb7b54768cc"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:14.732047 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.732028    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e76779fa-5562-44e6-abbd-9fb7b54768cc" (UID: "e76779fa-5562-44e6-abbd-9fb7b54768cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:14.733677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.733646    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76779fa-5562-44e6-abbd-9fb7b54768cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e76779fa-5562-44e6-abbd-9fb7b54768cc" (UID: "e76779fa-5562-44e6-abbd-9fb7b54768cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:21:14.733784 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.733681    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76779fa-5562-44e6-abbd-9fb7b54768cc-kube-api-access-5zn94" (OuterVolumeSpecName: "kube-api-access-5zn94") pod "e76779fa-5562-44e6-abbd-9fb7b54768cc" (UID: "e76779fa-5562-44e6-abbd-9fb7b54768cc"). InnerVolumeSpecName "kube-api-access-5zn94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:21:14.832217 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.832191    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76779fa-5562-44e6-abbd-9fb7b54768cc-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:14.832217 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.832214    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:14.832364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.832223    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:14.832364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.832232    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e76779fa-5562-44e6-abbd-9fb7b54768cc-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:14.832364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:14.832241    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zn94\" (UniqueName: \"kubernetes.io/projected/e76779fa-5562-44e6-abbd-9fb7b54768cc-kube-api-access-5zn94\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:15.462345 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.462313    2578 generic.go:358] "Generic (PLEG): container finished" podID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerID="a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930" exitCode=0
Apr 16 16:21:15.462805 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.462380    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"
Apr 16 16:21:15.462805 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.462387    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerDied","Data":"a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930"}
Apr 16 16:21:15.462805 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.462431    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n" event={"ID":"e76779fa-5562-44e6-abbd-9fb7b54768cc","Type":"ContainerDied","Data":"e979e6c79911304968a644cdd7062843914e5b6c4f5e062d1d242f10cf647221"}
Apr 16 16:21:15.462805 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.462483    2578 scope.go:117] "RemoveContainer" containerID="a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930"
Apr 16 16:21:15.471422 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.471277    2578 scope.go:117] "RemoveContainer" containerID="98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad"
Apr 16 16:21:15.478891 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.478873    2578 scope.go:117] "RemoveContainer" containerID="722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6"
Apr 16 16:21:15.485326 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.485304    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"]
Apr 16 16:21:15.487639 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.487623    2578 scope.go:117] "RemoveContainer" containerID="a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930"
Apr 16 16:21:15.487929 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:21:15.487911    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930\": container with ID starting with a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930 not found: ID does not exist" containerID="a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930"
Apr 16 16:21:15.488003 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.487943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930"} err="failed to get container status \"a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930\": rpc error: code = NotFound desc = could not find container \"a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930\": container with ID starting with a9f88b696cc12b117aeb0a565bf4d3960d783e5e19ae7ed4f6d38de3d7ac7930 not found: ID does not exist"
Apr 16 16:21:15.488003 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.487968    2578 scope.go:117] "RemoveContainer" containerID="98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad"
Apr 16 16:21:15.488211 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:21:15.488194    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad\": container with ID starting with 98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad not found: ID does not exist" containerID="98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad"
Apr 16 16:21:15.488251 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.488217    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad"} err="failed to get container status \"98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad\": rpc error: code = NotFound desc = could not find container \"98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad\": container with ID starting with 98043d7110e7504c3613b007c41dcbb8bfbf6c99013c2e1666a8f0c363d146ad not found: ID does not exist"
Apr 16 16:21:15.488251 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.488231    2578 scope.go:117] "RemoveContainer" containerID="722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6"
Apr 16 16:21:15.488395 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.488373    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-7d484wp86n"]
Apr 16 16:21:15.488459 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:21:15.488429    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6\": container with ID starting with 722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6 not found: ID does not exist" containerID="722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6"
Apr 16 16:21:15.488515 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:15.488469    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6"} err="failed to get container status \"722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6\": rpc error: code = NotFound desc = could not find container \"722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6\": container with ID starting with 722521f2bc0e8177bc10838c3d7432526d02f91300344c17827cf3dd3a4b64b6 not found: ID does not exist"
Apr 16 16:21:17.107787 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:17.107745    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" path="/var/lib/kubelet/pods/e76779fa-5562-44e6-abbd-9fb7b54768cc/volumes"
Apr 16 16:21:18.105708 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.105671    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"]
Apr 16 16:21:18.106437 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106411    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="main"
Apr 16 16:21:18.106754 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106730    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="main"
Apr 16 16:21:18.106886 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106779    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="tokenizer"
Apr 16 16:21:18.106886 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106789    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="tokenizer"
Apr 16 16:21:18.106886 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106808    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="storage-initializer"
Apr 16 16:21:18.106886 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106817    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="storage-initializer"
Apr 16 16:21:18.107120 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106933    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="main"
Apr 16 16:21:18.107120 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.106953    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="e76779fa-5562-44e6-abbd-9fb7b54768cc" containerName="tokenizer"
Apr 16 16:21:18.111898 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.111879    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.114567 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.114543    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Apr 16 16:21:18.114685 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.114602    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 16 16:21:18.115824 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.115802    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2fdlg\""
Apr 16 16:21:18.115932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.115826    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-5fq82\""
Apr 16 16:21:18.115932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.115874    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 16 16:21:18.128024 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.128000    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"]
Apr 16 16:21:18.157492 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.157435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.157642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.157593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.157642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.157621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.157642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.157640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.157818 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.157745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd4edd8-f76c-47e1-b7ff-c68769905012-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.157818 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.157809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4t8\" (UniqueName: \"kubernetes.io/projected/9fd4edd8-f76c-47e1-b7ff-c68769905012-kube-api-access-kn4t8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.258734 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.258707    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd4edd8-f76c-47e1-b7ff-c68769905012-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.258895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.258755    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4t8\" (UniqueName: \"kubernetes.io/projected/9fd4edd8-f76c-47e1-b7ff-c68769905012-kube-api-access-kn4t8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.258895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.258779    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.258895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.258825    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.258895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.258844    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.258895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.258870    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.259266 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.259241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.259361 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.259260    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.259361 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.259318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.259438 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.259326    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.261368 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.261351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd4edd8-f76c-47e1-b7ff-c68769905012-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.268391 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.268370    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4t8\" (UniqueName: \"kubernetes.io/projected/9fd4edd8-f76c-47e1-b7ff-c68769905012-kube-api-access-kn4t8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.422593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.422521    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:18.565082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:18.565062    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"]
Apr 16 16:21:18.567609 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:21:18.567579    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd4edd8_f76c_47e1_b7ff_c68769905012.slice/crio-ac70580d69879229fb2e301ec034886ae89c0f580fb7b80a5d6f024ae3583025 WatchSource:0}: Error finding container ac70580d69879229fb2e301ec034886ae89c0f580fb7b80a5d6f024ae3583025: Status 404 returned error can't find the container with id ac70580d69879229fb2e301ec034886ae89c0f580fb7b80a5d6f024ae3583025
Apr 16 16:21:19.118133 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:19.118107    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:21:19.118765 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:19.118742    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:21:19.482341 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:19.482309    2578 generic.go:358] "Generic (PLEG): container finished" podID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerID="e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e" exitCode=0
Apr 16 16:21:19.482512 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:19.482403    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" event={"ID":"9fd4edd8-f76c-47e1-b7ff-c68769905012","Type":"ContainerDied","Data":"e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e"}
Apr 16 16:21:19.482512 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:19.482439    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" event={"ID":"9fd4edd8-f76c-47e1-b7ff-c68769905012","Type":"ContainerStarted","Data":"ac70580d69879229fb2e301ec034886ae89c0f580fb7b80a5d6f024ae3583025"}
Apr 16 16:21:20.491419 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:20.491378    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" event={"ID":"9fd4edd8-f76c-47e1-b7ff-c68769905012","Type":"ContainerStarted","Data":"dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a"}
Apr 16 16:21:20.491419 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:20.491421    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" event={"ID":"9fd4edd8-f76c-47e1-b7ff-c68769905012","Type":"ContainerStarted","Data":"00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272"}
Apr 16 16:21:20.491888 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:20.491593    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:20.515592 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:20.515538    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" podStartSLOduration=2.515520154 podStartE2EDuration="2.515520154s" podCreationTimestamp="2026-04-16 16:21:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:21:20.511065033 +0000 UTC m=+1201.907149496" watchObservedRunningTime="2026-04-16 16:21:20.515520154 +0000 UTC m=+1201.911604574"
Apr 16 16:21:28.422764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:28.422727    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:28.423316 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:28.422777    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:28.425586 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:28.425562    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:28.525854 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:28.525828    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:49.530977 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:49.530949    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:50.917688 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:50.917650    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"]
Apr 16 16:21:50.918178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:50.918109    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="main" containerID="cri-o://00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272" gracePeriod=30
Apr 16 16:21:50.918250 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:50.918211    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="tokenizer" containerID="cri-o://dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a" gracePeriod=30
Apr 16 16:21:51.619533 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:51.619495    2578 generic.go:358] "Generic (PLEG): container finished" podID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerID="00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272" exitCode=0
Apr 16 16:21:51.619715 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:51.619567    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" event={"ID":"9fd4edd8-f76c-47e1-b7ff-c68769905012","Type":"ContainerDied","Data":"00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272"}
Apr 16 16:21:52.064314 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.064293    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:52.231953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.231924    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-cache\") pod \"9fd4edd8-f76c-47e1-b7ff-c68769905012\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") "
Apr 16 16:21:52.232097 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.231991    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd4edd8-f76c-47e1-b7ff-c68769905012-tls-certs\") pod \"9fd4edd8-f76c-47e1-b7ff-c68769905012\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") "
Apr 16 16:21:52.232097 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232049    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-kserve-provision-location\") pod \"9fd4edd8-f76c-47e1-b7ff-c68769905012\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") "
Apr 16 16:21:52.232097 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232066    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn4t8\" (UniqueName: \"kubernetes.io/projected/9fd4edd8-f76c-47e1-b7ff-c68769905012-kube-api-access-kn4t8\") pod \"9fd4edd8-f76c-47e1-b7ff-c68769905012\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") "
Apr 16 16:21:52.232097 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232084    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-uds\") pod \"9fd4edd8-f76c-47e1-b7ff-c68769905012\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") "
Apr 16 16:21:52.232308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232116    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-tmp\") pod \"9fd4edd8-f76c-47e1-b7ff-c68769905012\" (UID: \"9fd4edd8-f76c-47e1-b7ff-c68769905012\") "
Apr 16 16:21:52.232308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232221    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "9fd4edd8-f76c-47e1-b7ff-c68769905012" (UID: "9fd4edd8-f76c-47e1-b7ff-c68769905012"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:52.232415 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232377    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:52.232415 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232385    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9fd4edd8-f76c-47e1-b7ff-c68769905012" (UID: "9fd4edd8-f76c-47e1-b7ff-c68769905012"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:52.232604 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232576    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "9fd4edd8-f76c-47e1-b7ff-c68769905012" (UID: "9fd4edd8-f76c-47e1-b7ff-c68769905012"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:52.232967 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.232943    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fd4edd8-f76c-47e1-b7ff-c68769905012" (UID: "9fd4edd8-f76c-47e1-b7ff-c68769905012"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:21:52.234214 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.234191    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fd4edd8-f76c-47e1-b7ff-c68769905012-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fd4edd8-f76c-47e1-b7ff-c68769905012" (UID: "9fd4edd8-f76c-47e1-b7ff-c68769905012"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:21:52.234306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.234285    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd4edd8-f76c-47e1-b7ff-c68769905012-kube-api-access-kn4t8" (OuterVolumeSpecName: "kube-api-access-kn4t8") pod "9fd4edd8-f76c-47e1-b7ff-c68769905012" (UID: "9fd4edd8-f76c-47e1-b7ff-c68769905012"). InnerVolumeSpecName "kube-api-access-kn4t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:21:52.332929 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.332907    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:52.332929 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.332929    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4t8\" (UniqueName: \"kubernetes.io/projected/9fd4edd8-f76c-47e1-b7ff-c68769905012-kube-api-access-kn4t8\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:52.333058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.332939    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:52.333058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.332948    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fd4edd8-f76c-47e1-b7ff-c68769905012-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:52.333058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.332957    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fd4edd8-f76c-47e1-b7ff-c68769905012-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:21:52.625381 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.625315    2578 generic.go:358] "Generic (PLEG): container finished" podID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerID="dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a" exitCode=0
Apr 16 16:21:52.625508 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.625389    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"
Apr 16 16:21:52.625508 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.625392    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" event={"ID":"9fd4edd8-f76c-47e1-b7ff-c68769905012","Type":"ContainerDied","Data":"dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a"}
Apr 16 16:21:52.625508 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.625428    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg" event={"ID":"9fd4edd8-f76c-47e1-b7ff-c68769905012","Type":"ContainerDied","Data":"ac70580d69879229fb2e301ec034886ae89c0f580fb7b80a5d6f024ae3583025"}
Apr 16 16:21:52.625508 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.625459    2578 scope.go:117] "RemoveContainer" containerID="dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a"
Apr 16 16:21:52.635498 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.635481    2578 scope.go:117] "RemoveContainer" containerID="00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272"
Apr 16 16:21:52.643973 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.643960    2578 scope.go:117] "RemoveContainer" containerID="e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e"
Apr 16 16:21:52.648279 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.648256    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"]
Apr 16 16:21:52.652768 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.652748    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-7df4bf9f6jfg"]
Apr 16 16:21:52.652912 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.652901    2578 scope.go:117] "RemoveContainer" containerID="dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a"
Apr 16 16:21:52.653165 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:21:52.653147    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a\": container with ID starting with dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a not found: ID does not exist" containerID="dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a"
Apr 16 16:21:52.653229 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.653169    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a"} err="failed to get container status \"dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a\": rpc error: code = NotFound desc = could not find container \"dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a\": container with ID starting with dcec3efa463614fabcb49dbd6922d8951bf76b2e4f5ac1181c32255c747bc17a not found: ID does not exist"
Apr 16 16:21:52.653229 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.653184    2578 scope.go:117] "RemoveContainer" containerID="00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272"
Apr 16 16:21:52.653384 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:21:52.653370    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272\": container with ID starting with 00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272 not found: ID does not exist" containerID="00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272"
Apr 16 16:21:52.653418 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.653389    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272"} err="failed to get container status \"00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272\": rpc error: code = NotFound desc = could not find container \"00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272\": container with ID starting with 00baf59f0845a08dbbcd1199c1770ef13416252421ccc5c7813767e8ff898272 not found: ID does not exist"
Apr 16 16:21:52.653418 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.653402    2578 scope.go:117] "RemoveContainer" containerID="e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e"
Apr 16 16:21:52.653618 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:21:52.653605    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e\": container with ID starting with e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e not found: ID does not exist" containerID="e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e"
Apr 16 16:21:52.653656 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:52.653622    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e"} err="failed to get container status \"e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e\": rpc error: code = NotFound desc = could not find container \"e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e\": container with ID starting with e6e8b84414476ad5e3f983ae1e2eb9c88178c018bb8567fdd7aaed97b1013a1e not found: ID does not exist"
Apr 16 16:21:53.107854 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:21:53.107819    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" path="/var/lib/kubelet/pods/9fd4edd8-f76c-47e1-b7ff-c68769905012/volumes"
Apr 16 16:22:00.790017 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.789943    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"]
Apr 16 16:22:00.790677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790573    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="storage-initializer"
Apr 16 16:22:00.790677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790595    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="storage-initializer"
Apr 16 16:22:00.790677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790616    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="tokenizer"
Apr 16 16:22:00.790677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790624    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="tokenizer"
Apr 16 16:22:00.790677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790646    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="main"
Apr 16 16:22:00.790677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790654    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="main"
Apr 16 16:22:00.791017 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790742    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="main"
Apr 16 16:22:00.791017 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.790757    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="9fd4edd8-f76c-47e1-b7ff-c68769905012" containerName="tokenizer"
Apr 16 16:22:00.795532 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.795511    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.798269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.798250    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 16 16:22:00.799319 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.799301    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Apr 16 16:22:00.799415 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.799322    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 16 16:22:00.799415 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.799360    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2fdlg\""
Apr 16 16:22:00.804489 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.804469    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"]
Apr 16 16:22:00.895826 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.895793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-dshm\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.895958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.895835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-home\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.895958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.895861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa971d0-0c43-469f-b18c-c5ae25b7573a-tls-certs\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.895958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.895937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.896086 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.895976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2x5\" (UniqueName: \"kubernetes.io/projected/daa971d0-0c43-469f-b18c-c5ae25b7573a-kube-api-access-9d2x5\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.896086 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.896061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-model-cache\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.996838 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.996810    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-model-cache\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.996954 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.996855    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-dshm\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.996954 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.996902    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-home\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.996954 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.996921    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa971d0-0c43-469f-b18c-c5ae25b7573a-tls-certs\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.997070 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.996960    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.997070 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.996994    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2x5\" (UniqueName: \"kubernetes.io/projected/daa971d0-0c43-469f-b18c-c5ae25b7573a-kube-api-access-9d2x5\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.997238 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.997216    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-model-cache\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.997358 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.997333    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-home\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.997495 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.997341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.999296 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.999275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-dshm\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:00.999414 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:00.999396    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa971d0-0c43-469f-b18c-c5ae25b7573a-tls-certs\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:01.006252 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.006233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2x5\" (UniqueName: \"kubernetes.io/projected/daa971d0-0c43-469f-b18c-c5ae25b7573a-kube-api-access-9d2x5\") pod \"precise-prefix-cache-test-kserve-d67dfdb78-4nk8w\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:01.106977 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.106897    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:01.112527 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.112500    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"]
Apr 16 16:22:01.118696 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.118678    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.121192 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.121171    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-mknsz\""
Apr 16 16:22:01.125353 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.125329    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"]
Apr 16 16:22:01.249828 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.249798    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"]
Apr 16 16:22:01.250753 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:22:01.250716    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa971d0_0c43_469f_b18c_c5ae25b7573a.slice/crio-d59d4938784a88a47fa826b598623cdd34955fb32c54175c1bf03d47ee4f00a6 WatchSource:0}: Error finding container d59d4938784a88a47fa826b598623cdd34955fb32c54175c1bf03d47ee4f00a6: Status 404 returned error can't find the container with id d59d4938784a88a47fa826b598623cdd34955fb32c54175c1bf03d47ee4f00a6
Apr 16 16:22:01.299530 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.299503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.299642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.299616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.299703 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.299670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lghfm\" (UniqueName: \"kubernetes.io/projected/b3d1e012-4233-4255-bd2d-a576245e673d-kube-api-access-lghfm\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.299767 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.299724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.299813 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.299777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d1e012-4233-4255-bd2d-a576245e673d-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.299862 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.299824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.400803 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.400724    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d1e012-4233-4255-bd2d-a576245e673d-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.400803 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.400782    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.400836    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.400887    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.400921    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lghfm\" (UniqueName: \"kubernetes.io/projected/b3d1e012-4233-4255-bd2d-a576245e673d-kube-api-access-lghfm\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.400955    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401315 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.401287    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401401 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.401344    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401401 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.401352    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.401658 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.401639    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.403470 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.403437    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d1e012-4233-4255-bd2d-a576245e673d-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.409270 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.409244    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lghfm\" (UniqueName: \"kubernetes.io/projected/b3d1e012-4233-4255-bd2d-a576245e673d-kube-api-access-lghfm\") pod \"precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.450505 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.450477    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:01.662372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.662282    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" event={"ID":"daa971d0-0c43-469f-b18c-c5ae25b7573a","Type":"ContainerStarted","Data":"447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8"}
Apr 16 16:22:01.662372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.662319    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" event={"ID":"daa971d0-0c43-469f-b18c-c5ae25b7573a","Type":"ContainerStarted","Data":"d59d4938784a88a47fa826b598623cdd34955fb32c54175c1bf03d47ee4f00a6"}
Apr 16 16:22:01.789500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:01.789469    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"]
Apr 16 16:22:01.790859 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:22:01.790831    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d1e012_4233_4255_bd2d_a576245e673d.slice/crio-f768a1b3b630e7110ce0d1f7e5ff2ae9f7c0aa620ebebf405499900d37a94aab WatchSource:0}: Error finding container f768a1b3b630e7110ce0d1f7e5ff2ae9f7c0aa620ebebf405499900d37a94aab: Status 404 returned error can't find the container with id f768a1b3b630e7110ce0d1f7e5ff2ae9f7c0aa620ebebf405499900d37a94aab
Apr 16 16:22:02.669928 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:02.669586    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerStarted","Data":"6e5d5761bae0b789022582a105deeea55f13a7a99110d0a2c037914f697c2dc6"}
Apr 16 16:22:02.669928 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:02.669638    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerStarted","Data":"f768a1b3b630e7110ce0d1f7e5ff2ae9f7c0aa620ebebf405499900d37a94aab"}
Apr 16 16:22:03.675536 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:03.675500    2578 generic.go:358] "Generic (PLEG): container finished" podID="b3d1e012-4233-4255-bd2d-a576245e673d" containerID="6e5d5761bae0b789022582a105deeea55f13a7a99110d0a2c037914f697c2dc6" exitCode=0
Apr 16 16:22:03.675966 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:03.675603    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerDied","Data":"6e5d5761bae0b789022582a105deeea55f13a7a99110d0a2c037914f697c2dc6"}
Apr 16 16:22:04.685001 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:04.684956    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerStarted","Data":"3c7a5db970ce0da3b4f25ac5d593df0780201fb5dfecf5b73518116e6ef292a2"}
Apr 16 16:22:04.685001 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:04.684997    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerStarted","Data":"ca76793fb7a92347a69e21c088491f7fe19a1ba542dd7f27e1963a566926dff1"}
Apr 16 16:22:04.685859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:04.685084    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:04.711854 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:04.711787    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" podStartSLOduration=3.711767451 podStartE2EDuration="3.711767451s" podCreationTimestamp="2026-04-16 16:22:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:22:04.707866562 +0000 UTC m=+1246.103950997" watchObservedRunningTime="2026-04-16 16:22:04.711767451 +0000 UTC m=+1246.107851872"
Apr 16 16:22:05.690487 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:05.690437    2578 generic.go:358] "Generic (PLEG): container finished" podID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerID="447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8" exitCode=0
Apr 16 16:22:05.690844 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:05.690516    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" event={"ID":"daa971d0-0c43-469f-b18c-c5ae25b7573a","Type":"ContainerDied","Data":"447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8"}
Apr 16 16:22:07.701439 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:07.701401    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" event={"ID":"daa971d0-0c43-469f-b18c-c5ae25b7573a","Type":"ContainerStarted","Data":"1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba"}
Apr 16 16:22:07.721102 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:07.721057    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" podStartSLOduration=6.628742147 podStartE2EDuration="7.721046105s" podCreationTimestamp="2026-04-16 16:22:00 +0000 UTC" firstStartedPulling="2026-04-16 16:22:05.691767741 +0000 UTC m=+1247.087852139" lastFinishedPulling="2026-04-16 16:22:06.784071685 +0000 UTC m=+1248.180156097" observedRunningTime="2026-04-16 16:22:07.719754256 +0000 UTC m=+1249.115838677" watchObservedRunningTime="2026-04-16 16:22:07.721046105 +0000 UTC m=+1249.117130526"
Apr 16 16:22:11.107728 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.107700    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:11.107728 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.107735    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:11.120367 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.120345    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:11.451429 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.451349    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:11.451429 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.451389    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:11.452545 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:22:11.452518    2578 logging.go:55] [core] [Channel #41 SubChannel #42]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.53:9003", ServerName: "10.134.0.53:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.53:9003: connect: connection refused"
Apr 16 16:22:11.454065 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.454038    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:11.722564 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.722539    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:11.732702 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:11.732680    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:12.452467 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:12.452395    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.53:9003\" within 1s: context deadline exceeded"
Apr 16 16:22:21.451494 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:22:21.451438    2578 logging.go:55] [core] [Channel #43 SubChannel #44]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.53:9003", ServerName: "10.134.0.53:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.53:9003: connect: connection refused"
Apr 16 16:22:22.451052 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:22.451009    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.53:9003\" within 1s: context deadline exceeded"
Apr 16 16:22:42.726170 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:42.726138    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:43.746018 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:43.745973    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"]
Apr 16 16:22:43.746994 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:43.746934    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" podUID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerName="main" containerID="cri-o://1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba" gracePeriod=30
Apr 16 16:22:43.750009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:43.749955    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"]
Apr 16 16:22:43.750393 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:43.750343    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="main" containerID="cri-o://ca76793fb7a92347a69e21c088491f7fe19a1ba542dd7f27e1963a566926dff1" gracePeriod=30
Apr 16 16:22:43.750393 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:43.750361    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="tokenizer" containerID="cri-o://3c7a5db970ce0da3b4f25ac5d593df0780201fb5dfecf5b73518116e6ef292a2" gracePeriod=30
Apr 16 16:22:44.024424 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.024396    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:44.096441 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.096398    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-kserve-provision-location\") pod \"daa971d0-0c43-469f-b18c-c5ae25b7573a\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") "
Apr 16 16:22:44.096668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.096491    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa971d0-0c43-469f-b18c-c5ae25b7573a-tls-certs\") pod \"daa971d0-0c43-469f-b18c-c5ae25b7573a\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") "
Apr 16 16:22:44.096668 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.096535    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d2x5\" (UniqueName: \"kubernetes.io/projected/daa971d0-0c43-469f-b18c-c5ae25b7573a-kube-api-access-9d2x5\") pod \"daa971d0-0c43-469f-b18c-c5ae25b7573a\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") "
Apr 16 16:22:44.096797 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.096733    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-model-cache\") pod \"daa971d0-0c43-469f-b18c-c5ae25b7573a\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") "
Apr 16 16:22:44.096797 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.096771    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-home\") pod \"daa971d0-0c43-469f-b18c-c5ae25b7573a\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") "
Apr 16 16:22:44.096906 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.096807    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-dshm\") pod \"daa971d0-0c43-469f-b18c-c5ae25b7573a\" (UID: \"daa971d0-0c43-469f-b18c-c5ae25b7573a\") "
Apr 16 16:22:44.096984 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.096957    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-model-cache" (OuterVolumeSpecName: "model-cache") pod "daa971d0-0c43-469f-b18c-c5ae25b7573a" (UID: "daa971d0-0c43-469f-b18c-c5ae25b7573a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:44.097088 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.097062    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-home" (OuterVolumeSpecName: "home") pod "daa971d0-0c43-469f-b18c-c5ae25b7573a" (UID: "daa971d0-0c43-469f-b18c-c5ae25b7573a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:44.097199 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.097176    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:44.097199 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.097195    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:44.098968 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.098942    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa971d0-0c43-469f-b18c-c5ae25b7573a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "daa971d0-0c43-469f-b18c-c5ae25b7573a" (UID: "daa971d0-0c43-469f-b18c-c5ae25b7573a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:22:44.099066 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.098962    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-dshm" (OuterVolumeSpecName: "dshm") pod "daa971d0-0c43-469f-b18c-c5ae25b7573a" (UID: "daa971d0-0c43-469f-b18c-c5ae25b7573a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:44.099191 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.099169    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa971d0-0c43-469f-b18c-c5ae25b7573a-kube-api-access-9d2x5" (OuterVolumeSpecName: "kube-api-access-9d2x5") pod "daa971d0-0c43-469f-b18c-c5ae25b7573a" (UID: "daa971d0-0c43-469f-b18c-c5ae25b7573a"). InnerVolumeSpecName "kube-api-access-9d2x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:22:44.160269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.160209    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "daa971d0-0c43-469f-b18c-c5ae25b7573a" (UID: "daa971d0-0c43-469f-b18c-c5ae25b7573a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:44.198652 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.198604    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:44.198652 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.198651    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa971d0-0c43-469f-b18c-c5ae25b7573a-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:44.198866 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.198667    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2x5\" (UniqueName: \"kubernetes.io/projected/daa971d0-0c43-469f-b18c-c5ae25b7573a-kube-api-access-9d2x5\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:44.198866 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.198679    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa971d0-0c43-469f-b18c-c5ae25b7573a-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:44.863264 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.863227    2578 generic.go:358] "Generic (PLEG): container finished" podID="b3d1e012-4233-4255-bd2d-a576245e673d" containerID="ca76793fb7a92347a69e21c088491f7fe19a1ba542dd7f27e1963a566926dff1" exitCode=0
Apr 16 16:22:44.863759 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.863305    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerDied","Data":"ca76793fb7a92347a69e21c088491f7fe19a1ba542dd7f27e1963a566926dff1"}
Apr 16 16:22:44.864842 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.864818    2578 generic.go:358] "Generic (PLEG): container finished" podID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerID="1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba" exitCode=0
Apr 16 16:22:44.864962 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.864861    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" event={"ID":"daa971d0-0c43-469f-b18c-c5ae25b7573a","Type":"ContainerDied","Data":"1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba"}
Apr 16 16:22:44.864962 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.864885    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w" event={"ID":"daa971d0-0c43-469f-b18c-c5ae25b7573a","Type":"ContainerDied","Data":"d59d4938784a88a47fa826b598623cdd34955fb32c54175c1bf03d47ee4f00a6"}
Apr 16 16:22:44.864962 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.864906    2578 scope.go:117] "RemoveContainer" containerID="1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba"
Apr 16 16:22:44.864962 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.864919    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"
Apr 16 16:22:44.874862 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.874842    2578 scope.go:117] "RemoveContainer" containerID="447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8"
Apr 16 16:22:44.889966 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.889945    2578 scope.go:117] "RemoveContainer" containerID="1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba"
Apr 16 16:22:44.890296 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:22:44.890266    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba\": container with ID starting with 1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba not found: ID does not exist" containerID="1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba"
Apr 16 16:22:44.890385 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.890306    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba"} err="failed to get container status \"1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba\": rpc error: code = NotFound desc = could not find container \"1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba\": container with ID starting with 1b8702bb365cec94530d54cc63e496f3bfea5cef6db54f1e5d539cdd9649ceba not found: ID does not exist"
Apr 16 16:22:44.890385 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.890327    2578 scope.go:117] "RemoveContainer" containerID="447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8"
Apr 16 16:22:44.890636 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:22:44.890614    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8\": container with ID starting with 447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8 not found: ID does not exist" containerID="447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8"
Apr 16 16:22:44.890687 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.890646    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8"} err="failed to get container status \"447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8\": rpc error: code = NotFound desc = could not find container \"447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8\": container with ID starting with 447c7e758b094e3f096562b6f9c3563d566b412d917adb50c87c2dfd5d4137f8 not found: ID does not exist"
Apr 16 16:22:44.898322 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.898289    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"]
Apr 16 16:22:44.906109 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:44.906084    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-d67dfdb78-4nk8w"]
Apr 16 16:22:45.107985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:45.107955    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa971d0-0c43-469f-b18c-c5ae25b7573a" path="/var/lib/kubelet/pods/daa971d0-0c43-469f-b18c-c5ae25b7573a/volumes"
Apr 16 16:22:45.871576 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:45.871537    2578 generic.go:358] "Generic (PLEG): container finished" podID="b3d1e012-4233-4255-bd2d-a576245e673d" containerID="3c7a5db970ce0da3b4f25ac5d593df0780201fb5dfecf5b73518116e6ef292a2" exitCode=0
Apr 16 16:22:45.872004 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:45.871615    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerDied","Data":"3c7a5db970ce0da3b4f25ac5d593df0780201fb5dfecf5b73518116e6ef292a2"}
Apr 16 16:22:45.908219 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:45.908193    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:46.017746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.017623    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-uds\") pod \"b3d1e012-4233-4255-bd2d-a576245e673d\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") "
Apr 16 16:22:46.017746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.017686    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-tmp\") pod \"b3d1e012-4233-4255-bd2d-a576245e673d\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") "
Apr 16 16:22:46.017746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.017731    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-cache\") pod \"b3d1e012-4233-4255-bd2d-a576245e673d\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") "
Apr 16 16:22:46.018060 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.017803    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d1e012-4233-4255-bd2d-a576245e673d-tls-certs\") pod \"b3d1e012-4233-4255-bd2d-a576245e673d\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") "
Apr 16 16:22:46.018060 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.017856    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-kserve-provision-location\") pod \"b3d1e012-4233-4255-bd2d-a576245e673d\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") "
Apr 16 16:22:46.018060 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.017888    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lghfm\" (UniqueName: \"kubernetes.io/projected/b3d1e012-4233-4255-bd2d-a576245e673d-kube-api-access-lghfm\") pod \"b3d1e012-4233-4255-bd2d-a576245e673d\" (UID: \"b3d1e012-4233-4255-bd2d-a576245e673d\") "
Apr 16 16:22:46.018060 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.017916    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b3d1e012-4233-4255-bd2d-a576245e673d" (UID: "b3d1e012-4233-4255-bd2d-a576245e673d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:46.018237 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.018084    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "b3d1e012-4233-4255-bd2d-a576245e673d" (UID: "b3d1e012-4233-4255-bd2d-a576245e673d"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:46.018237 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.018216    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:46.018237 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.018235    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:46.018396 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.018212    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b3d1e012-4233-4255-bd2d-a576245e673d" (UID: "b3d1e012-4233-4255-bd2d-a576245e673d"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:46.018609 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.018581    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3d1e012-4233-4255-bd2d-a576245e673d" (UID: "b3d1e012-4233-4255-bd2d-a576245e673d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:22:46.020216 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.020188    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d1e012-4233-4255-bd2d-a576245e673d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3d1e012-4233-4255-bd2d-a576245e673d" (UID: "b3d1e012-4233-4255-bd2d-a576245e673d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:22:46.020320 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.020222    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d1e012-4233-4255-bd2d-a576245e673d-kube-api-access-lghfm" (OuterVolumeSpecName: "kube-api-access-lghfm") pod "b3d1e012-4233-4255-bd2d-a576245e673d" (UID: "b3d1e012-4233-4255-bd2d-a576245e673d"). InnerVolumeSpecName "kube-api-access-lghfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:22:46.118967 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.118920    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:46.118967 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.118958    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d1e012-4233-4255-bd2d-a576245e673d-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:46.118967 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.118970    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3d1e012-4233-4255-bd2d-a576245e673d-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:46.118967 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.118980    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lghfm\" (UniqueName: \"kubernetes.io/projected/b3d1e012-4233-4255-bd2d-a576245e673d-kube-api-access-lghfm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:22:46.879165 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.879127    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws" event={"ID":"b3d1e012-4233-4255-bd2d-a576245e673d","Type":"ContainerDied","Data":"f768a1b3b630e7110ce0d1f7e5ff2ae9f7c0aa620ebebf405499900d37a94aab"}
Apr 16 16:22:46.879165 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.879159    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"
Apr 16 16:22:46.879751 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.879188    2578 scope.go:117] "RemoveContainer" containerID="3c7a5db970ce0da3b4f25ac5d593df0780201fb5dfecf5b73518116e6ef292a2"
Apr 16 16:22:46.889631 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.889467    2578 scope.go:117] "RemoveContainer" containerID="ca76793fb7a92347a69e21c088491f7fe19a1ba542dd7f27e1963a566926dff1"
Apr 16 16:22:46.906672 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.906623    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"]
Apr 16 16:22:46.907006 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.906969    2578 scope.go:117] "RemoveContainer" containerID="6e5d5761bae0b789022582a105deeea55f13a7a99110d0a2c037914f697c2dc6"
Apr 16 16:22:46.912387 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:46.912354    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-54f664cfzkwws"]
Apr 16 16:22:47.109145 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:47.109101    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" path="/var/lib/kubelet/pods/b3d1e012-4233-4255-bd2d-a576245e673d/volumes"
Apr 16 16:22:52.170362 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.170315    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"]
Apr 16 16:22:52.171034 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171011    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="tokenizer"
Apr 16 16:22:52.171123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171038    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="tokenizer"
Apr 16 16:22:52.171123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171054    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerName="storage-initializer"
Apr 16 16:22:52.171123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171063    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerName="storage-initializer"
Apr 16 16:22:52.171123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171086    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="main"
Apr 16 16:22:52.171123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171094    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="main"
Apr 16 16:22:52.171123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171111    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerName="main"
Apr 16 16:22:52.171123 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171120    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerName="main"
Apr 16 16:22:52.171516 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171133    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="storage-initializer"
Apr 16 16:22:52.171516 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171142    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="storage-initializer"
Apr 16 16:22:52.171516 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171247    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="main"
Apr 16 16:22:52.171516 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171265    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="daa971d0-0c43-469f-b18c-c5ae25b7573a" containerName="main"
Apr 16 16:22:52.171516 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.171276    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b3d1e012-4233-4255-bd2d-a576245e673d" containerName="tokenizer"
Apr 16 16:22:52.176296 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.176270    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.179114 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.179089    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 16 16:22:52.179390 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.179359    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2fdlg\""
Apr 16 16:22:52.179512 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.179230    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Apr 16 16:22:52.179573 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.179555    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 16 16:22:52.184299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.184271    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"]
Apr 16 16:22:52.278467 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.278417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrjv\" (UniqueName: \"kubernetes.io/projected/26049e0e-116e-4ef4-9fb1-f183fa8871af-kube-api-access-clrjv\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.278711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.278499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26049e0e-116e-4ef4-9fb1-f183fa8871af-tls-certs\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.278711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.278541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-home\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.278711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.278684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-model-cache\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.278877 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.278712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-dshm\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.278877 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.278734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-kserve-provision-location\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.379749 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.379690    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26049e0e-116e-4ef4-9fb1-f183fa8871af-tls-certs\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.379749 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.379759    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-home\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.380057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.379824    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-model-cache\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.380057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.379844    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-dshm\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.380057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.379878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-kserve-provision-location\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.380057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.379949    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-clrjv\" (UniqueName: \"kubernetes.io/projected/26049e0e-116e-4ef4-9fb1-f183fa8871af-kube-api-access-clrjv\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.380252 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.380193    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-home\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.380337 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.380312    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-model-cache\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.380403 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.380362    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-kserve-provision-location\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.382500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.382468    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-dshm\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.382773 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.382754    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26049e0e-116e-4ef4-9fb1-f183fa8871af-tls-certs\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.389184 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.389157    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrjv\" (UniqueName: \"kubernetes.io/projected/26049e0e-116e-4ef4-9fb1-f183fa8871af-kube-api-access-clrjv\") pod \"stop-feature-test-kserve-85568b7f4f-q2cfr\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.488851 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.488811    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:22:52.641222 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.641143    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"]
Apr 16 16:22:52.643968 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:22:52.643924    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26049e0e_116e_4ef4_9fb1_f183fa8871af.slice/crio-3d64f7de0061766ea36debc9cb405fb279446b3b15c1aed254b55ea6cb2ac055 WatchSource:0}: Error finding container 3d64f7de0061766ea36debc9cb405fb279446b3b15c1aed254b55ea6cb2ac055: Status 404 returned error can't find the container with id 3d64f7de0061766ea36debc9cb405fb279446b3b15c1aed254b55ea6cb2ac055
Apr 16 16:22:52.912220 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.912085    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" event={"ID":"26049e0e-116e-4ef4-9fb1-f183fa8871af","Type":"ContainerStarted","Data":"a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f"}
Apr 16 16:22:52.912220 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:52.912144    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" event={"ID":"26049e0e-116e-4ef4-9fb1-f183fa8871af","Type":"ContainerStarted","Data":"3d64f7de0061766ea36debc9cb405fb279446b3b15c1aed254b55ea6cb2ac055"}
Apr 16 16:22:57.941606 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:57.941564    2578 generic.go:358] "Generic (PLEG): container finished" podID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerID="a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f" exitCode=0
Apr 16 16:22:57.942025 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:22:57.941633    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" event={"ID":"26049e0e-116e-4ef4-9fb1-f183fa8871af","Type":"ContainerDied","Data":"a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f"}
Apr 16 16:23:26.113541 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:26.113488    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" event={"ID":"26049e0e-116e-4ef4-9fb1-f183fa8871af","Type":"ContainerStarted","Data":"dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013"}
Apr 16 16:23:26.140298 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:26.140229    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podStartSLOduration=6.725785024 podStartE2EDuration="34.140211278s" podCreationTimestamp="2026-04-16 16:22:52 +0000 UTC" firstStartedPulling="2026-04-16 16:22:57.942853029 +0000 UTC m=+1299.338937427" lastFinishedPulling="2026-04-16 16:23:25.357279283 +0000 UTC m=+1326.753363681" observedRunningTime="2026-04-16 16:23:26.137195915 +0000 UTC m=+1327.533280315" watchObservedRunningTime="2026-04-16 16:23:26.140211278 +0000 UTC m=+1327.536295698"
Apr 16 16:23:30.162641 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.162543    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"]
Apr 16 16:23:30.193762 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.193710    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"]
Apr 16 16:23:30.193976 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.193839    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.196732 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.196697    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
Apr 16 16:23:30.196732 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.196697    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-55f7ae4a-epp-sa-dockercfg-tqzfn\""
Apr 16 16:23:30.378299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.378236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.378646 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.378325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.378646 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.378401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6h9q\" (UniqueName: \"kubernetes.io/projected/643722a1-0743-4f4b-8273-cc8c016fe352-kube-api-access-b6h9q\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.378646 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.378429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643722a1-0743-4f4b-8273-cc8c016fe352-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.378646 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.378465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.378646 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.378577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.479577 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.479508    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.479855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.479648    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.479855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.479748    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b6h9q\" (UniqueName: \"kubernetes.io/projected/643722a1-0743-4f4b-8273-cc8c016fe352-kube-api-access-b6h9q\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.479855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.479789    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643722a1-0743-4f4b-8273-cc8c016fe352-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.479855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.479816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.480081 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.479869    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.480081 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.480032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-tmp\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.480176 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.480131    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-kserve-provision-location\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.480226 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.480204    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-uds\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.480323 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.480297    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-cache\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.482752 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.482729    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643722a1-0743-4f4b-8273-cc8c016fe352-tls-certs\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.488757 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.488720    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6h9q\" (UniqueName: \"kubernetes.io/projected/643722a1-0743-4f4b-8273-cc8c016fe352-kube-api-access-b6h9q\") pod \"llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") " pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.505713 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.505657    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:30.683320 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:30.683266    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"]
Apr 16 16:23:30.684965 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:23:30.684921    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643722a1_0743_4f4b_8273_cc8c016fe352.slice/crio-aa912743482ba51cd1fa3df4792052a425eda879cb03c3d4d6cafe875c61d2a8 WatchSource:0}: Error finding container aa912743482ba51cd1fa3df4792052a425eda879cb03c3d4d6cafe875c61d2a8: Status 404 returned error can't find the container with id aa912743482ba51cd1fa3df4792052a425eda879cb03c3d4d6cafe875c61d2a8
Apr 16 16:23:31.139192 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:31.139057    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerStarted","Data":"2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca"}
Apr 16 16:23:31.139192 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:31.139115    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerStarted","Data":"aa912743482ba51cd1fa3df4792052a425eda879cb03c3d4d6cafe875c61d2a8"}
Apr 16 16:23:32.145234 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:32.145187    2578 generic.go:358] "Generic (PLEG): container finished" podID="643722a1-0743-4f4b-8273-cc8c016fe352" containerID="2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca" exitCode=0
Apr 16 16:23:32.145716 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:32.145276    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerDied","Data":"2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca"}
Apr 16 16:23:32.489182 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:32.489138    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:23:32.489527 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:32.489509    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:23:32.491072 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:32.491020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:23:33.151678 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:33.151641    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerStarted","Data":"91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619"}
Apr 16 16:23:33.151678 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:33.151686    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerStarted","Data":"5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198"}
Apr 16 16:23:33.152130 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:33.151749    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:33.179571 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:33.179507    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" podStartSLOduration=3.179487058 podStartE2EDuration="3.179487058s" podCreationTimestamp="2026-04-16 16:23:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:23:33.174528176 +0000 UTC m=+1334.570612596" watchObservedRunningTime="2026-04-16 16:23:33.179487058 +0000 UTC m=+1334.575571478"
Apr 16 16:23:40.506074 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:40.505997    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:40.506074 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:40.506074    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:40.507620 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:40.507566    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.55:8082/healthz\": dial tcp 10.134.0.55:8082: connect: connection refused"
Apr 16 16:23:42.489940 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:42.489879    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:23:50.508139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:50.508105    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:50.509595 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:50.509570    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:23:52.489677 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:23:52.489632    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:24:02.489863 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:24:02.489814    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:24:11.255515 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:24:11.255480    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:24:12.489932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:24:12.489879    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:24:22.490127 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:24:22.490074    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:24:32.489830 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:24:32.489779    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:24:42.489985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:24:42.489933    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:24:52.489615 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:24:52.489555    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:25:02.490375 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:02.490261    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Apr 16 16:25:12.499629 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:12.499598    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:25:12.507632 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:12.507598    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:25:13.951582 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:13.951547    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"]
Apr 16 16:25:13.952056 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:13.951846    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main" containerID="cri-o://dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013" gracePeriod=30
Apr 16 16:25:44.229105 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.229079    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-85568b7f4f-q2cfr_26049e0e-116e-4ef4-9fb1-f183fa8871af/main/0.log"
Apr 16 16:25:44.229521 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.229481    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:25:44.278546 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.278516    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26049e0e-116e-4ef4-9fb1-f183fa8871af-tls-certs\") pod \"26049e0e-116e-4ef4-9fb1-f183fa8871af\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") "
Apr 16 16:25:44.278727 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.278571    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-kserve-provision-location\") pod \"26049e0e-116e-4ef4-9fb1-f183fa8871af\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") "
Apr 16 16:25:44.278727 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.278654    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clrjv\" (UniqueName: \"kubernetes.io/projected/26049e0e-116e-4ef4-9fb1-f183fa8871af-kube-api-access-clrjv\") pod \"26049e0e-116e-4ef4-9fb1-f183fa8871af\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") "
Apr 16 16:25:44.278727 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.278683    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-home\") pod \"26049e0e-116e-4ef4-9fb1-f183fa8871af\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") "
Apr 16 16:25:44.278727 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.278698    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-dshm\") pod \"26049e0e-116e-4ef4-9fb1-f183fa8871af\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") "
Apr 16 16:25:44.278727 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.278719    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-model-cache\") pod \"26049e0e-116e-4ef4-9fb1-f183fa8871af\" (UID: \"26049e0e-116e-4ef4-9fb1-f183fa8871af\") "
Apr 16 16:25:44.279164 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.279140    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-home" (OuterVolumeSpecName: "home") pod "26049e0e-116e-4ef4-9fb1-f183fa8871af" (UID: "26049e0e-116e-4ef4-9fb1-f183fa8871af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:44.279250 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.279232    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-model-cache" (OuterVolumeSpecName: "model-cache") pod "26049e0e-116e-4ef4-9fb1-f183fa8871af" (UID: "26049e0e-116e-4ef4-9fb1-f183fa8871af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:44.281400 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.281348    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26049e0e-116e-4ef4-9fb1-f183fa8871af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26049e0e-116e-4ef4-9fb1-f183fa8871af" (UID: "26049e0e-116e-4ef4-9fb1-f183fa8871af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:25:44.281400 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.281350    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-dshm" (OuterVolumeSpecName: "dshm") pod "26049e0e-116e-4ef4-9fb1-f183fa8871af" (UID: "26049e0e-116e-4ef4-9fb1-f183fa8871af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:44.281602 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.281530    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26049e0e-116e-4ef4-9fb1-f183fa8871af-kube-api-access-clrjv" (OuterVolumeSpecName: "kube-api-access-clrjv") pod "26049e0e-116e-4ef4-9fb1-f183fa8871af" (UID: "26049e0e-116e-4ef4-9fb1-f183fa8871af"). InnerVolumeSpecName "kube-api-access-clrjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:25:44.337517 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.337476    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26049e0e-116e-4ef4-9fb1-f183fa8871af" (UID: "26049e0e-116e-4ef4-9fb1-f183fa8871af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:44.379705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.379664    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:44.379705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.379702    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26049e0e-116e-4ef4-9fb1-f183fa8871af-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:44.379705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.379713    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:44.379938 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.379723    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrjv\" (UniqueName: \"kubernetes.io/projected/26049e0e-116e-4ef4-9fb1-f183fa8871af-kube-api-access-clrjv\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:44.379938 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.379731    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:44.379938 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.379740    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26049e0e-116e-4ef4-9fb1-f183fa8871af-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:44.751301 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.751276    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-85568b7f4f-q2cfr_26049e0e-116e-4ef4-9fb1-f183fa8871af/main/0.log"
Apr 16 16:25:44.751631 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.751607    2578 generic.go:358] "Generic (PLEG): container finished" podID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerID="dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013" exitCode=137
Apr 16 16:25:44.751752 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.751661    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" event={"ID":"26049e0e-116e-4ef4-9fb1-f183fa8871af","Type":"ContainerDied","Data":"dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013"}
Apr 16 16:25:44.751752 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.751680    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"
Apr 16 16:25:44.751752 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.751698    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr" event={"ID":"26049e0e-116e-4ef4-9fb1-f183fa8871af","Type":"ContainerDied","Data":"3d64f7de0061766ea36debc9cb405fb279446b3b15c1aed254b55ea6cb2ac055"}
Apr 16 16:25:44.751752 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.751721    2578 scope.go:117] "RemoveContainer" containerID="dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013"
Apr 16 16:25:44.772428 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.772404    2578 scope.go:117] "RemoveContainer" containerID="a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f"
Apr 16 16:25:44.776336 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.776312    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"]
Apr 16 16:25:44.779117 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.779095    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-85568b7f4f-q2cfr"]
Apr 16 16:25:44.838018 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.837998    2578 scope.go:117] "RemoveContainer" containerID="dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013"
Apr 16 16:25:44.838360 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:25:44.838338    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013\": container with ID starting with dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013 not found: ID does not exist" containerID="dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013"
Apr 16 16:25:44.838482 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.838370    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013"} err="failed to get container status \"dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013\": rpc error: code = NotFound desc = could not find container \"dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013\": container with ID starting with dbb8e160400fbddbbeb0f82ac9e95c9738318ef79dc61c8fcbcf63af6062d013 not found: ID does not exist"
Apr 16 16:25:44.838482 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.838392    2578 scope.go:117] "RemoveContainer" containerID="a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f"
Apr 16 16:25:44.838723 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:25:44.838705    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f\": container with ID starting with a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f not found: ID does not exist" containerID="a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f"
Apr 16 16:25:44.838764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:44.838731    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f"} err="failed to get container status \"a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f\": rpc error: code = NotFound desc = could not find container \"a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f\": container with ID starting with a6605641619d59a04da70868a9e0fbebb832220c80d9bdd057c5e32161f66c4f not found: ID does not exist"
Apr 16 16:25:45.109990 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:45.109514    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" path="/var/lib/kubelet/pods/26049e0e-116e-4ef4-9fb1-f183fa8871af/volumes"
Apr 16 16:25:57.132296 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:57.132265    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"]
Apr 16 16:25:57.132886 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:57.132580    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="main" containerID="cri-o://5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198" gracePeriod=30
Apr 16 16:25:57.132886 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:57.132645    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="tokenizer" containerID="cri-o://91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619" gracePeriod=30
Apr 16 16:25:57.805544 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:57.805509    2578 generic.go:358] "Generic (PLEG): container finished" podID="643722a1-0743-4f4b-8273-cc8c016fe352" containerID="5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198" exitCode=0
Apr 16 16:25:57.805740 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:57.805601    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerDied","Data":"5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198"}
Apr 16 16:25:58.288822 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.288801    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:25:58.394770 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.394702    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-tmp\") pod \"643722a1-0743-4f4b-8273-cc8c016fe352\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") "
Apr 16 16:25:58.394770 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.394732    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-kserve-provision-location\") pod \"643722a1-0743-4f4b-8273-cc8c016fe352\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") "
Apr 16 16:25:58.394770 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.394762    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6h9q\" (UniqueName: \"kubernetes.io/projected/643722a1-0743-4f4b-8273-cc8c016fe352-kube-api-access-b6h9q\") pod \"643722a1-0743-4f4b-8273-cc8c016fe352\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") "
Apr 16 16:25:58.394975 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.394783    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643722a1-0743-4f4b-8273-cc8c016fe352-tls-certs\") pod \"643722a1-0743-4f4b-8273-cc8c016fe352\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") "
Apr 16 16:25:58.394975 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.394862    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-uds\") pod \"643722a1-0743-4f4b-8273-cc8c016fe352\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") "
Apr 16 16:25:58.394975 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.394896    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-cache\") pod \"643722a1-0743-4f4b-8273-cc8c016fe352\" (UID: \"643722a1-0743-4f4b-8273-cc8c016fe352\") "
Apr 16 16:25:58.395108 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.395043    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "643722a1-0743-4f4b-8273-cc8c016fe352" (UID: "643722a1-0743-4f4b-8273-cc8c016fe352"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:58.395178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.395130    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:58.395248 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.395225    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "643722a1-0743-4f4b-8273-cc8c016fe352" (UID: "643722a1-0743-4f4b-8273-cc8c016fe352"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:58.395303 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.395269    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "643722a1-0743-4f4b-8273-cc8c016fe352" (UID: "643722a1-0743-4f4b-8273-cc8c016fe352"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:58.395596 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.395570    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "643722a1-0743-4f4b-8273-cc8c016fe352" (UID: "643722a1-0743-4f4b-8273-cc8c016fe352"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:25:58.396966 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.396946    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643722a1-0743-4f4b-8273-cc8c016fe352-kube-api-access-b6h9q" (OuterVolumeSpecName: "kube-api-access-b6h9q") pod "643722a1-0743-4f4b-8273-cc8c016fe352" (UID: "643722a1-0743-4f4b-8273-cc8c016fe352"). InnerVolumeSpecName "kube-api-access-b6h9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:25:58.397060 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.397034    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643722a1-0743-4f4b-8273-cc8c016fe352-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "643722a1-0743-4f4b-8273-cc8c016fe352" (UID: "643722a1-0743-4f4b-8273-cc8c016fe352"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:25:58.495670 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.495645    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:58.495670 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.495667    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:58.495803 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.495676    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/643722a1-0743-4f4b-8273-cc8c016fe352-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:58.495803 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.495686    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6h9q\" (UniqueName: \"kubernetes.io/projected/643722a1-0743-4f4b-8273-cc8c016fe352-kube-api-access-b6h9q\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:58.495803 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.495696    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643722a1-0743-4f4b-8273-cc8c016fe352-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:25:58.811093 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.811066    2578 generic.go:358] "Generic (PLEG): container finished" podID="643722a1-0743-4f4b-8273-cc8c016fe352" containerID="91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619" exitCode=0
Apr 16 16:25:58.811224 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.811154    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerDied","Data":"91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619"}
Apr 16 16:25:58.811224 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.811165    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"
Apr 16 16:25:58.811224 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.811188    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn" event={"ID":"643722a1-0743-4f4b-8273-cc8c016fe352","Type":"ContainerDied","Data":"aa912743482ba51cd1fa3df4792052a425eda879cb03c3d4d6cafe875c61d2a8"}
Apr 16 16:25:58.811224 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.811205    2578 scope.go:117] "RemoveContainer" containerID="91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619"
Apr 16 16:25:58.824146 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.824122    2578 scope.go:117] "RemoveContainer" containerID="5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198"
Apr 16 16:25:58.831647 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.831627    2578 scope.go:117] "RemoveContainer" containerID="2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca"
Apr 16 16:25:58.840364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.840333    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"]
Apr 16 16:25:58.841210 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.841170    2578 scope.go:117] "RemoveContainer" containerID="91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619"
Apr 16 16:25:58.841500 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:25:58.841476    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619\": container with ID starting with 91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619 not found: ID does not exist" containerID="91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619"
Apr 16 16:25:58.841597 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.841506    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619"} err="failed to get container status \"91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619\": rpc error: code = NotFound desc = could not find container \"91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619\": container with ID starting with 91d33abf322741746c6e89e7c830c765408861bb119827da5ef945184bf55619 not found: ID does not exist"
Apr 16 16:25:58.841597 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.841522    2578 scope.go:117] "RemoveContainer" containerID="5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198"
Apr 16 16:25:58.841789 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:25:58.841771    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198\": container with ID starting with 5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198 not found: ID does not exist" containerID="5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198"
Apr 16 16:25:58.841834 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.841793    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198"} err="failed to get container status \"5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198\": rpc error: code = NotFound desc = could not find container \"5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198\": container with ID starting with 5a82b28d58461942305fcf28fa11ff2b529d0740ad391cc8e16f2e53e33c2198 not found: ID does not exist"
Apr 16 16:25:58.841834 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.841807    2578 scope.go:117] "RemoveContainer" containerID="2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca"
Apr 16 16:25:58.842051 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:25:58.842031    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca\": container with ID starting with 2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca not found: ID does not exist" containerID="2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca"
Apr 16 16:25:58.842104 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.842055    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca"} err="failed to get container status \"2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca\": rpc error: code = NotFound desc = could not find container \"2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca\": container with ID starting with 2c753cbf8bed404e7cea282b0dda909a88b3463419ed8b2defcb2f39bcc9d4ca not found: ID does not exist"
Apr 16 16:25:58.843818 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:58.843800    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcdde380eaa9fe1facad32d45131f9e34d-kserve-router-schebfcrn"]
Apr 16 16:25:59.108029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:25:59.107964    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" path="/var/lib/kubelet/pods/643722a1-0743-4f4b-8273-cc8c016fe352/volumes"
Apr 16 16:26:12.577832 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.577789    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"]
Apr 16 16:26:12.578488 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578470    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="storage-initializer"
Apr 16 16:26:12.578557 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578491    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="storage-initializer"
Apr 16 16:26:12.578557 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578506    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="tokenizer"
Apr 16 16:26:12.578557 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578515    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="tokenizer"
Apr 16 16:26:12.578557 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578536    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main"
Apr 16 16:26:12.578557 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578544    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main"
Apr 16 16:26:12.578711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578561    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="storage-initializer"
Apr 16 16:26:12.578711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578570    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="storage-initializer"
Apr 16 16:26:12.578711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578593    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="main"
Apr 16 16:26:12.578711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578602    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="main"
Apr 16 16:26:12.578711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578697    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="tokenizer"
Apr 16 16:26:12.578711 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578709    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="26049e0e-116e-4ef4-9fb1-f183fa8871af" containerName="main"
Apr 16 16:26:12.578882 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.578721    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="643722a1-0743-4f4b-8273-cc8c016fe352" containerName="main"
Apr 16 16:26:12.582562 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.582537    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.585192 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.585155    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2fdlg\""
Apr 16 16:26:12.585192 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.585180    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 16 16:26:12.585409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.585234    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 16 16:26:12.585409 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.585293    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Apr 16 16:26:12.589997 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.589964    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"]
Apr 16 16:26:12.703290 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.703249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-home\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.703527 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.703496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.703649 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.703628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4js\" (UniqueName: \"kubernetes.io/projected/a53f9672-d823-4386-8098-1522849cb2ed-kube-api-access-wr4js\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.703714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.703674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-model-cache\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.703714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.703700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f9672-d823-4386-8098-1522849cb2ed-tls-certs\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.703802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.703767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-dshm\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.804832 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.804799    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-home\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.804866    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.804896    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4js\" (UniqueName: \"kubernetes.io/projected/a53f9672-d823-4386-8098-1522849cb2ed-kube-api-access-wr4js\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.804916    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-model-cache\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.804936    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f9672-d823-4386-8098-1522849cb2ed-tls-certs\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.804955    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-dshm\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805315 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.805258    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-home\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.805332    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-model-cache\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.805425 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.805380    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.807826 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.807805    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-dshm\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.808072 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.808055    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f9672-d823-4386-8098-1522849cb2ed-tls-certs\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.812775 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.812755    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4js\" (UniqueName: \"kubernetes.io/projected/a53f9672-d823-4386-8098-1522849cb2ed-kube-api-access-wr4js\") pod \"custom-route-timeout-test-kserve-55c77bfb77-pqzxf\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:12.851054 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.850989    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"]
Apr 16 16:26:12.854997 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.854978    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:12.857475 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.857430    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-b8mrc\""
Apr 16 16:26:12.866290 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.866268    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"]
Apr 16 16:26:12.894859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:12.894833    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:13.007182 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.007139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75v7z\" (UniqueName: \"kubernetes.io/projected/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kube-api-access-75v7z\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.007344 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.007197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.007344 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.007224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.007344 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.007279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.007559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.007376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.007559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.007408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.029637 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.029564    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"]
Apr 16 16:26:13.031990 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:26:13.031963    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53f9672_d823_4386_8098_1522849cb2ed.slice/crio-8888bafc147d9d82fcc5fb518e57f259cb8c8ac71838441eae13f3d5f116f918 WatchSource:0}: Error finding container 8888bafc147d9d82fcc5fb518e57f259cb8c8ac71838441eae13f3d5f116f918: Status 404 returned error can't find the container with id 8888bafc147d9d82fcc5fb518e57f259cb8c8ac71838441eae13f3d5f116f918
Apr 16 16:26:13.033866 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.033850    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:26:13.108131 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108049    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-75v7z\" (UniqueName: \"kubernetes.io/projected/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kube-api-access-75v7z\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108131 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108096    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108324 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108217    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108324 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108303    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108476 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108349    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108476 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108375    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108476 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108402    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108661 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108572    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108662    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.108743 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.108710    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.110831 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.110812    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.116583 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.116553    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75v7z\" (UniqueName: \"kubernetes.io/projected/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kube-api-access-75v7z\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.167194 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.167152    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:13.336972 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.336939    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"]
Apr 16 16:26:13.338215 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:26:13.338182    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd1505c_eac3_48da_a4a3_f49d52b9500e.slice/crio-a62b6cf0edf0c1b54159e293ffbc2c4ac9e2dff21ede8b8fecffc1b16275c803 WatchSource:0}: Error finding container a62b6cf0edf0c1b54159e293ffbc2c4ac9e2dff21ede8b8fecffc1b16275c803: Status 404 returned error can't find the container with id a62b6cf0edf0c1b54159e293ffbc2c4ac9e2dff21ede8b8fecffc1b16275c803
Apr 16 16:26:13.872971 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.872931    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerStarted","Data":"022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16"}
Apr 16 16:26:13.873427 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.872978    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerStarted","Data":"a62b6cf0edf0c1b54159e293ffbc2c4ac9e2dff21ede8b8fecffc1b16275c803"}
Apr 16 16:26:13.874521 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.874491    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" event={"ID":"a53f9672-d823-4386-8098-1522849cb2ed","Type":"ContainerStarted","Data":"4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f"}
Apr 16 16:26:13.874609 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:13.874531    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" event={"ID":"a53f9672-d823-4386-8098-1522849cb2ed","Type":"ContainerStarted","Data":"8888bafc147d9d82fcc5fb518e57f259cb8c8ac71838441eae13f3d5f116f918"}
Apr 16 16:26:14.879944 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:14.879908    2578 generic.go:358] "Generic (PLEG): container finished" podID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerID="022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16" exitCode=0
Apr 16 16:26:14.880403 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:14.879973    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerDied","Data":"022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16"}
Apr 16 16:26:15.890926 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:15.890886    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerStarted","Data":"9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b"}
Apr 16 16:26:15.891349 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:15.890935    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerStarted","Data":"d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6"}
Apr 16 16:26:15.891349 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:15.891039    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:15.915552 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:15.915496    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" podStartSLOduration=3.915474784 podStartE2EDuration="3.915474784s" podCreationTimestamp="2026-04-16 16:26:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:26:15.910427313 +0000 UTC m=+1497.306511775" watchObservedRunningTime="2026-04-16 16:26:15.915474784 +0000 UTC m=+1497.311559206"
Apr 16 16:26:17.903164 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:17.903132    2578 generic.go:358] "Generic (PLEG): container finished" podID="a53f9672-d823-4386-8098-1522849cb2ed" containerID="4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f" exitCode=0
Apr 16 16:26:17.903587 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:17.903205    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" event={"ID":"a53f9672-d823-4386-8098-1522849cb2ed","Type":"ContainerDied","Data":"4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f"}
Apr 16 16:26:18.909596 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:18.909559    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" event={"ID":"a53f9672-d823-4386-8098-1522849cb2ed","Type":"ContainerStarted","Data":"fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba"}
Apr 16 16:26:18.930819 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:18.930761    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podStartSLOduration=6.930742163 podStartE2EDuration="6.930742163s" podCreationTimestamp="2026-04-16 16:26:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:26:18.92783616 +0000 UTC m=+1500.323920579" watchObservedRunningTime="2026-04-16 16:26:18.930742163 +0000 UTC m=+1500.326826584"
Apr 16 16:26:19.157722 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:19.157686    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:26:19.159186 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:19.159162    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:26:22.896014 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:22.895980    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:22.896472 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:22.896024    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:26:22.897393 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:22.897366    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:26:23.168040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:23.167944    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:23.168040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:23.167977    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:23.170864 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:23.170838    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:23.938551 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:23.938511    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:32.895951 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:32.895869    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:26:42.895271 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:42.895226    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:26:44.953630 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:44.953600    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:26:52.895892 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:26:52.895853    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:27:02.895671 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:02.895622    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:27:12.896031 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:12.895996    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:27:22.895608 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:22.895562    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:27:32.895299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:32.895252    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:27:42.895476 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:42.895424    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Apr 16 16:27:52.905416 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:52.905387    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:27:52.913148 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:52.913122    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:27:58.907263 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:58.907172    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"]
Apr 16 16:27:58.908308 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:58.908229    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="main" containerID="cri-o://d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6" gracePeriod=30
Apr 16 16:27:58.908967 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:58.908930    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="tokenizer" containerID="cri-o://9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b" gracePeriod=30
Apr 16 16:27:58.918240 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:58.918215    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"]
Apr 16 16:27:58.918604 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:58.918563    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main" containerID="cri-o://fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba" gracePeriod=30
Apr 16 16:27:59.359666 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:59.359632    2578 generic.go:358] "Generic (PLEG): container finished" podID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerID="d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6" exitCode=0
Apr 16 16:27:59.359861 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:27:59.359711    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerDied","Data":"d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6"}
Apr 16 16:28:00.069215 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.069191    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:28:00.158845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.158777    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-cache\") pod \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") "
Apr 16 16:28:00.158845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.158816    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-tmp\") pod \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") "
Apr 16 16:28:00.159045 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.158860    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-uds\") pod \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") "
Apr 16 16:28:00.159045 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.158990    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kserve-provision-location\") pod \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") "
Apr 16 16:28:00.159045 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159030    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "6cd1505c-eac3-48da-a4a3-f49d52b9500e" (UID: "6cd1505c-eac3-48da-a4a3-f49d52b9500e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:00.159215 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159082    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tls-certs\") pod \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") "
Apr 16 16:28:00.159215 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159094    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6cd1505c-eac3-48da-a4a3-f49d52b9500e" (UID: "6cd1505c-eac3-48da-a4a3-f49d52b9500e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:00.159215 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159108    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75v7z\" (UniqueName: \"kubernetes.io/projected/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kube-api-access-75v7z\") pod \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\" (UID: \"6cd1505c-eac3-48da-a4a3-f49d52b9500e\") "
Apr 16 16:28:00.159374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159210    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "6cd1505c-eac3-48da-a4a3-f49d52b9500e" (UID: "6cd1505c-eac3-48da-a4a3-f49d52b9500e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:00.159374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159345    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:00.159374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159358    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:00.159374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159368    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:00.159611 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.159590    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6cd1505c-eac3-48da-a4a3-f49d52b9500e" (UID: "6cd1505c-eac3-48da-a4a3-f49d52b9500e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:00.161295 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.161275    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kube-api-access-75v7z" (OuterVolumeSpecName: "kube-api-access-75v7z") pod "6cd1505c-eac3-48da-a4a3-f49d52b9500e" (UID: "6cd1505c-eac3-48da-a4a3-f49d52b9500e"). InnerVolumeSpecName "kube-api-access-75v7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:28:00.161384 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.161353    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cd1505c-eac3-48da-a4a3-f49d52b9500e" (UID: "6cd1505c-eac3-48da-a4a3-f49d52b9500e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:28:00.260175 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.260148    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:00.260307 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.260179    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cd1505c-eac3-48da-a4a3-f49d52b9500e-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:00.260307 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.260196    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75v7z\" (UniqueName: \"kubernetes.io/projected/6cd1505c-eac3-48da-a4a3-f49d52b9500e-kube-api-access-75v7z\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:00.365932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.365905    2578 generic.go:358] "Generic (PLEG): container finished" podID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerID="9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b" exitCode=0
Apr 16 16:28:00.366040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.365980    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerDied","Data":"9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b"}
Apr 16 16:28:00.366040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.365981    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"
Apr 16 16:28:00.366040 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.366016    2578 scope.go:117] "RemoveContainer" containerID="9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b"
Apr 16 16:28:00.366220 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.366007    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx" event={"ID":"6cd1505c-eac3-48da-a4a3-f49d52b9500e","Type":"ContainerDied","Data":"a62b6cf0edf0c1b54159e293ffbc2c4ac9e2dff21ede8b8fecffc1b16275c803"}
Apr 16 16:28:00.375328 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.375311    2578 scope.go:117] "RemoveContainer" containerID="d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6"
Apr 16 16:28:00.383211 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.383195    2578 scope.go:117] "RemoveContainer" containerID="022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16"
Apr 16 16:28:00.387718 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.387696    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"]
Apr 16 16:28:00.391035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.391014    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56dfc4zwdmx"]
Apr 16 16:28:00.391965 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.391948    2578 scope.go:117] "RemoveContainer" containerID="9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b"
Apr 16 16:28:00.392190 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:28:00.392175    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b\": container with ID starting with 9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b not found: ID does not exist" containerID="9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b"
Apr 16 16:28:00.392235 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.392199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b"} err="failed to get container status \"9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b\": rpc error: code = NotFound desc = could not find container \"9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b\": container with ID starting with 9c2e707b014e7a5187d0a8b96cc6c2f3e94228c2702114d81d8ff2046c99cb5b not found: ID does not exist"
Apr 16 16:28:00.392235 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.392220    2578 scope.go:117] "RemoveContainer" containerID="d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6"
Apr 16 16:28:00.392412 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:28:00.392395    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6\": container with ID starting with d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6 not found: ID does not exist" containerID="d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6"
Apr 16 16:28:00.392472 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.392420    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6"} err="failed to get container status \"d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6\": rpc error: code = NotFound desc = could not find container \"d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6\": container with ID starting with d5826c1ea3e779f3a38ea424461acb8d715e687ad076ccbe645276c686be8df6 not found: ID does not exist"
Apr 16 16:28:00.392472 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.392437    2578 scope.go:117] "RemoveContainer" containerID="022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16"
Apr 16 16:28:00.392677 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:28:00.392655    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16\": container with ID starting with 022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16 not found: ID does not exist" containerID="022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16"
Apr 16 16:28:00.392766 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:00.392680    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16"} err="failed to get container status \"022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16\": rpc error: code = NotFound desc = could not find container \"022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16\": container with ID starting with 022876015fc6e50f93cd943a2d1dd7773418fbd4fb098dab42b5dec325a44d16 not found: ID does not exist"
Apr 16 16:28:01.108575 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:01.108544    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" path="/var/lib/kubelet/pods/6cd1505c-eac3-48da-a4a3-f49d52b9500e/volumes"
Apr 16 16:28:02.687260 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:02.687226    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-fc44f49f-m2vc9"]
Apr 16 16:28:02.687644 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:02.687542    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9" podUID="746f3884-ce3c-4397-b2d2-417c2c8782a5" containerName="manager" containerID="cri-o://1f1147d2f00ac5fcdb8c4ed565aecb625693195e2e44dddb56b2ab33908fe7ee" gracePeriod=30
Apr 16 16:28:12.418129 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.418093    2578 generic.go:358] "Generic (PLEG): container finished" podID="746f3884-ce3c-4397-b2d2-417c2c8782a5" containerID="1f1147d2f00ac5fcdb8c4ed565aecb625693195e2e44dddb56b2ab33908fe7ee" exitCode=0
Apr 16 16:28:12.420083 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.420042    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9" event={"ID":"746f3884-ce3c-4397-b2d2-417c2c8782a5","Type":"ContainerDied","Data":"1f1147d2f00ac5fcdb8c4ed565aecb625693195e2e44dddb56b2ab33908fe7ee"}
Apr 16 16:28:12.453626 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.453599    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:28:12.456796 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.456774    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert\") pod \"746f3884-ce3c-4397-b2d2-417c2c8782a5\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") "
Apr 16 16:28:12.456917 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.456837    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ng9n\" (UniqueName: \"kubernetes.io/projected/746f3884-ce3c-4397-b2d2-417c2c8782a5-kube-api-access-9ng9n\") pod \"746f3884-ce3c-4397-b2d2-417c2c8782a5\" (UID: \"746f3884-ce3c-4397-b2d2-417c2c8782a5\") "
Apr 16 16:28:12.458891 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.458862    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746f3884-ce3c-4397-b2d2-417c2c8782a5-kube-api-access-9ng9n" (OuterVolumeSpecName: "kube-api-access-9ng9n") pod "746f3884-ce3c-4397-b2d2-417c2c8782a5" (UID: "746f3884-ce3c-4397-b2d2-417c2c8782a5"). InnerVolumeSpecName "kube-api-access-9ng9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:28:12.459010 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.458909    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert" (OuterVolumeSpecName: "cert") pod "746f3884-ce3c-4397-b2d2-417c2c8782a5" (UID: "746f3884-ce3c-4397-b2d2-417c2c8782a5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:28:12.558158 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.558098    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/746f3884-ce3c-4397-b2d2-417c2c8782a5-cert\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:12.558158 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:12.558122    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ng9n\" (UniqueName: \"kubernetes.io/projected/746f3884-ce3c-4397-b2d2-417c2c8782a5-kube-api-access-9ng9n\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:13.425526 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:13.425433    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9"
Apr 16 16:28:13.425526 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:13.425477    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-fc44f49f-m2vc9" event={"ID":"746f3884-ce3c-4397-b2d2-417c2c8782a5","Type":"ContainerDied","Data":"3305d726a1053f9cfea982cc33ea70655c02bc52d8123e3923bcb31e8c4b2d16"}
Apr 16 16:28:13.425526 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:13.425524    2578 scope.go:117] "RemoveContainer" containerID="1f1147d2f00ac5fcdb8c4ed565aecb625693195e2e44dddb56b2ab33908fe7ee"
Apr 16 16:28:13.443482 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:13.443435    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-fc44f49f-m2vc9"]
Apr 16 16:28:13.446053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:13.446031    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-fc44f49f-m2vc9"]
Apr 16 16:28:15.108236 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:15.108200    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746f3884-ce3c-4397-b2d2-417c2c8782a5" path="/var/lib/kubelet/pods/746f3884-ce3c-4397-b2d2-417c2c8782a5/volumes"
Apr 16 16:28:29.196602 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.196577    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-55c77bfb77-pqzxf_a53f9672-d823-4386-8098-1522849cb2ed/main/0.log"
Apr 16 16:28:29.196924 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.196910    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:28:29.276702 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.276674    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f9672-d823-4386-8098-1522849cb2ed-tls-certs\") pod \"a53f9672-d823-4386-8098-1522849cb2ed\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") "
Apr 16 16:28:29.276845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.276706    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr4js\" (UniqueName: \"kubernetes.io/projected/a53f9672-d823-4386-8098-1522849cb2ed-kube-api-access-wr4js\") pod \"a53f9672-d823-4386-8098-1522849cb2ed\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") "
Apr 16 16:28:29.276845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.276742    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-model-cache\") pod \"a53f9672-d823-4386-8098-1522849cb2ed\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") "
Apr 16 16:28:29.276845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.276756    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-dshm\") pod \"a53f9672-d823-4386-8098-1522849cb2ed\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") "
Apr 16 16:28:29.276845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.276817    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-home\") pod \"a53f9672-d823-4386-8098-1522849cb2ed\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") "
Apr 16 16:28:29.276845 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.276834    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-kserve-provision-location\") pod \"a53f9672-d823-4386-8098-1522849cb2ed\" (UID: \"a53f9672-d823-4386-8098-1522849cb2ed\") "
Apr 16 16:28:29.277105 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.276994    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "a53f9672-d823-4386-8098-1522849cb2ed" (UID: "a53f9672-d823-4386-8098-1522849cb2ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:29.277164 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.277146    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-home" (OuterVolumeSpecName: "home") pod "a53f9672-d823-4386-8098-1522849cb2ed" (UID: "a53f9672-d823-4386-8098-1522849cb2ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:29.278976 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.278947    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53f9672-d823-4386-8098-1522849cb2ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a53f9672-d823-4386-8098-1522849cb2ed" (UID: "a53f9672-d823-4386-8098-1522849cb2ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:28:29.279063 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.279013    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-dshm" (OuterVolumeSpecName: "dshm") pod "a53f9672-d823-4386-8098-1522849cb2ed" (UID: "a53f9672-d823-4386-8098-1522849cb2ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:29.279063 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.279028    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53f9672-d823-4386-8098-1522849cb2ed-kube-api-access-wr4js" (OuterVolumeSpecName: "kube-api-access-wr4js") pod "a53f9672-d823-4386-8098-1522849cb2ed" (UID: "a53f9672-d823-4386-8098-1522849cb2ed"). InnerVolumeSpecName "kube-api-access-wr4js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:28:29.342107 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.342075    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a53f9672-d823-4386-8098-1522849cb2ed" (UID: "a53f9672-d823-4386-8098-1522849cb2ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:28:29.378295 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.378268    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:29.378399 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.378295    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:29.378399 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.378311    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f9672-d823-4386-8098-1522849cb2ed-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:29.378399 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.378326    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4js\" (UniqueName: \"kubernetes.io/projected/a53f9672-d823-4386-8098-1522849cb2ed-kube-api-access-wr4js\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:29.378399 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.378338    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:29.378399 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.378350    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53f9672-d823-4386-8098-1522849cb2ed-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:28:29.499035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.499015    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-55c77bfb77-pqzxf_a53f9672-d823-4386-8098-1522849cb2ed/main/0.log"
Apr 16 16:28:29.499341 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.499319    2578 generic.go:358] "Generic (PLEG): container finished" podID="a53f9672-d823-4386-8098-1522849cb2ed" containerID="fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba" exitCode=137
Apr 16 16:28:29.499407 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.499394    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"
Apr 16 16:28:29.499472 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.499403    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" event={"ID":"a53f9672-d823-4386-8098-1522849cb2ed","Type":"ContainerDied","Data":"fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba"}
Apr 16 16:28:29.499472 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.499440    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf" event={"ID":"a53f9672-d823-4386-8098-1522849cb2ed","Type":"ContainerDied","Data":"8888bafc147d9d82fcc5fb518e57f259cb8c8ac71838441eae13f3d5f116f918"}
Apr 16 16:28:29.499551 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.499482    2578 scope.go:117] "RemoveContainer" containerID="fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba"
Apr 16 16:28:29.518362 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.518344    2578 scope.go:117] "RemoveContainer" containerID="4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f"
Apr 16 16:28:29.524002 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.523977    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"]
Apr 16 16:28:29.528107 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.528088    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-55c77bfb77-pqzxf"]
Apr 16 16:28:29.529012 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.528999    2578 scope.go:117] "RemoveContainer" containerID="fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba"
Apr 16 16:28:29.529256 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:28:29.529237    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba\": container with ID starting with fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba not found: ID does not exist" containerID="fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba"
Apr 16 16:28:29.529326 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.529268    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba"} err="failed to get container status \"fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba\": rpc error: code = NotFound desc = could not find container \"fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba\": container with ID starting with fb02ae9abba28937afe40e2e2f87482a5d165a103b4d47a282b43b86f85fdeba not found: ID does not exist"
Apr 16 16:28:29.529326 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.529292    2578 scope.go:117] "RemoveContainer" containerID="4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f"
Apr 16 16:28:29.529566 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:28:29.529545    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f\": container with ID starting with 4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f not found: ID does not exist" containerID="4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f"
Apr 16 16:28:29.529669 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:29.529569    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f"} err="failed to get container status \"4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f\": rpc error: code = NotFound desc = could not find container \"4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f\": container with ID starting with 4f945222d81bf707e23f34d1648364ec9b55901b07cef0b2084a23abd9e4a00f not found: ID does not exist"
Apr 16 16:28:31.107783 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:31.107750    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53f9672-d823-4386-8098-1522849cb2ed" path="/var/lib/kubelet/pods/a53f9672-d823-4386-8098-1522849cb2ed/volumes"
Apr 16 16:28:39.852103 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852069    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"]
Apr 16 16:28:39.852652 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852636    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746f3884-ce3c-4397-b2d2-417c2c8782a5" containerName="manager"
Apr 16 16:28:39.852714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852656    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="746f3884-ce3c-4397-b2d2-417c2c8782a5" containerName="manager"
Apr 16 16:28:39.852714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852676    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="storage-initializer"
Apr 16 16:28:39.852714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852685    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="storage-initializer"
Apr 16 16:28:39.852714 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852711    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="tokenizer"
Apr 16 16:28:39.852837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852719    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="tokenizer"
Apr 16 16:28:39.852837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852731    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="storage-initializer"
Apr 16 16:28:39.852837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852739    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="storage-initializer"
Apr 16 16:28:39.852837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852754    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="main"
Apr 16 16:28:39.852837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852762    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="main"
Apr 16 16:28:39.852837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852772    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main"
Apr 16 16:28:39.852837 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main"
Apr 16 16:28:39.853082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852859    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="tokenizer"
Apr 16 16:28:39.853082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852873    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6cd1505c-eac3-48da-a4a3-f49d52b9500e" containerName="main"
Apr 16 16:28:39.853082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852884    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a53f9672-d823-4386-8098-1522849cb2ed" containerName="main"
Apr 16 16:28:39.853082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.852896    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="746f3884-ce3c-4397-b2d2-417c2c8782a5" containerName="manager"
Apr 16 16:28:39.856740 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.856720    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:39.860239 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.860218    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-x5mxx\""
Apr 16 16:28:39.860355 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.860257    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Apr 16 16:28:39.860355 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.860318    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Apr 16 16:28:39.860505 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.860218    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2fdlg\""
Apr 16 16:28:39.860505 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.860368    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Apr 16 16:28:39.866725 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.866705    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"]
Apr 16 16:28:39.958521 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.958489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:39.958629 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.958531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:39.958629 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.958558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spx76\" (UniqueName: \"kubernetes.io/projected/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kube-api-access-spx76\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:39.958709 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.958643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:39.958767 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.958747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:39.958803 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:39.958784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059501 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059474    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059503    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059523    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-spx76\" (UniqueName: \"kubernetes.io/projected/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kube-api-access-spx76\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059745 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059649    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059780    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059856 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059812    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059911 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059895    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.059962 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.059944    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.060086 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.060068    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.060179 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.060162    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.061990 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.061971    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.067297 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.067278    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spx76\" (UniqueName: \"kubernetes.io/projected/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kube-api-access-spx76\") pod \"router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.167545 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.167483    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:40.294865 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.294833    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"]
Apr 16 16:28:40.296413 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:28:40.296379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e857ccc_440e_4b23_9ff7_93cd5e1f0e88.slice/crio-5a4c8cc9e3d7f71d74c912c31c22392beb3afc467d928e6a3cff13beb9dcd418 WatchSource:0}: Error finding container 5a4c8cc9e3d7f71d74c912c31c22392beb3afc467d928e6a3cff13beb9dcd418: Status 404 returned error can't find the container with id 5a4c8cc9e3d7f71d74c912c31c22392beb3afc467d928e6a3cff13beb9dcd418
Apr 16 16:28:40.550172 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.550126    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerStarted","Data":"eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6"}
Apr 16 16:28:40.550172 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:40.550165    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerStarted","Data":"5a4c8cc9e3d7f71d74c912c31c22392beb3afc467d928e6a3cff13beb9dcd418"}
Apr 16 16:28:41.554775 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:41.554739    2578 generic.go:358] "Generic (PLEG): container finished" podID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerID="eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6" exitCode=0
Apr 16 16:28:41.555242 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:41.554830    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerDied","Data":"eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6"}
Apr 16 16:28:42.561245 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:42.561192    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerStarted","Data":"73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c"}
Apr 16 16:28:42.561245 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:42.561249    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerStarted","Data":"4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290"}
Apr 16 16:28:42.561667 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:42.561298    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:42.583958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:42.583915    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" podStartSLOduration=3.583900212 podStartE2EDuration="3.583900212s" podCreationTimestamp="2026-04-16 16:28:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:28:42.582263629 +0000 UTC m=+1643.978348055" watchObservedRunningTime="2026-04-16 16:28:42.583900212 +0000 UTC m=+1643.979984631"
Apr 16 16:28:50.167784 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:50.167755    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:50.168136 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:50.167902    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:50.170543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:50.170518    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:28:50.595125 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:28:50.595101    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:29:00.614465 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.614424    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"]
Apr 16 16:29:00.618888 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.618865    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.621877 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.621858    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Apr 16 16:29:00.638743 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.637507    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"]
Apr 16 16:29:00.719470 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.719412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.719645 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.719522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5356d2f-1700-48f6-beaf-f5bc3957d89d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.719645 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.719550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crq42\" (UniqueName: \"kubernetes.io/projected/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kube-api-access-crq42\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.719645 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.719571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.719645 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.719617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-home\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.719799 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.719652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.820850 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.820816    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.820887    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5356d2f-1700-48f6-beaf-f5bc3957d89d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.820915    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-crq42\" (UniqueName: \"kubernetes.io/projected/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kube-api-access-crq42\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.820934    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821029 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.820958    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-home\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821688 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.821572    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821688 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.821650    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821802 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.821773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-home\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.821929 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.821896    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.823648 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.823620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.824262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.824241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5356d2f-1700-48f6-beaf-f5bc3957d89d-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.828868 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.828842    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crq42\" (UniqueName: \"kubernetes.io/projected/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kube-api-access-crq42\") pod \"router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:00.930414 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:00.930361    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:01.058425 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:01.058395    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"]
Apr 16 16:29:01.060134 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:29:01.060106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5356d2f_1700_48f6_beaf_f5bc3957d89d.slice/crio-4441ec582e8916732d3e72f3e3e9a0d2fa8892034326510e3875c7391f0868ec WatchSource:0}: Error finding container 4441ec582e8916732d3e72f3e3e9a0d2fa8892034326510e3875c7391f0868ec: Status 404 returned error can't find the container with id 4441ec582e8916732d3e72f3e3e9a0d2fa8892034326510e3875c7391f0868ec
Apr 16 16:29:01.646716 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:01.646679    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" event={"ID":"c5356d2f-1700-48f6-beaf-f5bc3957d89d","Type":"ContainerStarted","Data":"e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568"}
Apr 16 16:29:01.646716 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:01.646719    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" event={"ID":"c5356d2f-1700-48f6-beaf-f5bc3957d89d","Type":"ContainerStarted","Data":"4441ec582e8916732d3e72f3e3e9a0d2fa8892034326510e3875c7391f0868ec"}
Apr 16 16:29:05.666693 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:05.666653    2578 generic.go:358] "Generic (PLEG): container finished" podID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerID="e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568" exitCode=0
Apr 16 16:29:05.667075 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:05.666728    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" event={"ID":"c5356d2f-1700-48f6-beaf-f5bc3957d89d","Type":"ContainerDied","Data":"e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568"}
Apr 16 16:29:06.673827 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:06.673792    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" event={"ID":"c5356d2f-1700-48f6-beaf-f5bc3957d89d","Type":"ContainerStarted","Data":"a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6"}
Apr 16 16:29:06.696518 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:06.696474    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podStartSLOduration=6.696460265 podStartE2EDuration="6.696460265s" podCreationTimestamp="2026-04-16 16:29:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:29:06.693397017 +0000 UTC m=+1668.089481430" watchObservedRunningTime="2026-04-16 16:29:06.696460265 +0000 UTC m=+1668.092544676"
Apr 16 16:29:10.930619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:10.930569    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:10.930619 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:10.930632    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:29:10.932062 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:10.932031    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:29:12.603659 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:12.603616    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:29:20.931176 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:20.931130    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:29:30.930932 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:30.930831    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:29:40.930793 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:40.930743    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:29:50.931429 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:29:50.931387    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:30:00.931459 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:00.931390    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:30:10.931624 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:10.931569    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:30:20.931466 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:20.931402    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:30:30.931385 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:30.931342    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Apr 16 16:30:40.940273 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:40.940241    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:30:40.947990 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:40.947970    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:30:48.342357 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:48.342315    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"]
Apr 16 16:30:48.342782 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:48.342752    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="main" containerID="cri-o://4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290" gracePeriod=30
Apr 16 16:30:48.342855 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:48.342808    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="tokenizer" containerID="cri-o://73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c" gracePeriod=30
Apr 16 16:30:49.118073 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.118041    2578 generic.go:358] "Generic (PLEG): container finished" podID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerID="4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290" exitCode=0
Apr 16 16:30:49.118278 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.118115    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerDied","Data":"4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290"}
Apr 16 16:30:49.489826 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.489805    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:30:49.555291 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555196    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kserve-provision-location\") pod \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") "
Apr 16 16:30:49.555418 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555359    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tls-certs\") pod \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") "
Apr 16 16:30:49.555418 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555390    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-cache\") pod \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") "
Apr 16 16:30:49.555519 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555440    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-uds\") pod \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") "
Apr 16 16:30:49.555554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555515    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spx76\" (UniqueName: \"kubernetes.io/projected/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kube-api-access-spx76\") pod \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") "
Apr 16 16:30:49.555554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555539    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-tmp\") pod \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\" (UID: \"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88\") "
Apr 16 16:30:49.555697 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555674    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" (UID: "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:30:49.555825 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555781    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" (UID: "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:30:49.555913 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555896    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:30:49.555970 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555921    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:30:49.555970 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.555938    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" (UID: "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:30:49.556076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.556058    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" (UID: "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:30:49.557503 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.557482    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" (UID: "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:30:49.557603 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.557584    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kube-api-access-spx76" (OuterVolumeSpecName: "kube-api-access-spx76") pod "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" (UID: "4e857ccc-440e-4b23-9ff7-93cd5e1f0e88"). InnerVolumeSpecName "kube-api-access-spx76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:30:49.657249 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.657182    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spx76\" (UniqueName: \"kubernetes.io/projected/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kube-api-access-spx76\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:30:49.657249 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.657206    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:30:49.657249 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.657216    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:30:49.657249 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:49.657225    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:30:50.123634 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.123591    2578 generic.go:358] "Generic (PLEG): container finished" podID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerID="73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c" exitCode=0
Apr 16 16:30:50.123784 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.123676    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"
Apr 16 16:30:50.123784 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.123678    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerDied","Data":"73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c"}
Apr 16 16:30:50.123784 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.123711    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz" event={"ID":"4e857ccc-440e-4b23-9ff7-93cd5e1f0e88","Type":"ContainerDied","Data":"5a4c8cc9e3d7f71d74c912c31c22392beb3afc467d928e6a3cff13beb9dcd418"}
Apr 16 16:30:50.123784 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.123728    2578 scope.go:117] "RemoveContainer" containerID="73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c"
Apr 16 16:30:50.134794 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.134772    2578 scope.go:117] "RemoveContainer" containerID="4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290"
Apr 16 16:30:50.143493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.143472    2578 scope.go:117] "RemoveContainer" containerID="eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6"
Apr 16 16:30:50.152405 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.152388    2578 scope.go:117] "RemoveContainer" containerID="73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c"
Apr 16 16:30:50.152825 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:30:50.152800    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c\": container with ID starting with 73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c not found: ID does not exist" containerID="73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c"
Apr 16 16:30:50.152989 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.152835    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c"} err="failed to get container status \"73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c\": rpc error: code = NotFound desc = could not find container \"73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c\": container with ID starting with 73f1b671bf096b262dbaf4873c21d84eebcc509e75e6fd9c9e670eae5815315c not found: ID does not exist"
Apr 16 16:30:50.152989 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.152858    2578 scope.go:117] "RemoveContainer" containerID="4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290"
Apr 16 16:30:50.153202 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:30:50.153173    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290\": container with ID starting with 4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290 not found: ID does not exist" containerID="4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290"
Apr 16 16:30:50.153318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.153210    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290"} err="failed to get container status \"4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290\": rpc error: code = NotFound desc = could not find container \"4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290\": container with ID starting with 4a17a64958689897e70f3f3f58efac41800a422a84a81dafe7559caf690ab290 not found: ID does not exist"
Apr 16 16:30:50.153318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.153235    2578 scope.go:117] "RemoveContainer" containerID="eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6"
Apr 16 16:30:50.153318 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.153252    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"]
Apr 16 16:30:50.153590 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:30:50.153568    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6\": container with ID starting with eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6 not found: ID does not exist" containerID="eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6"
Apr 16 16:30:50.153651 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.153599    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6"} err="failed to get container status \"eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6\": rpc error: code = NotFound desc = could not find container \"eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6\": container with ID starting with eaf3e13899b5e8c3bba1872bd183a5fc005bdf07faaf7a51cd37738744c0f7b6 not found: ID does not exist"
Apr 16 16:30:50.155708 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:50.155687    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-8cdf76d65-22vnz"]
Apr 16 16:30:51.107705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:30:51.107674    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" path="/var/lib/kubelet/pods/4e857ccc-440e-4b23-9ff7-93cd5e1f0e88/volumes"
Apr 16 16:31:12.384299 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384265    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"]
Apr 16 16:31:12.384731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384641    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="storage-initializer"
Apr 16 16:31:12.384731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384653    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="storage-initializer"
Apr 16 16:31:12.384731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384672    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="tokenizer"
Apr 16 16:31:12.384731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="tokenizer"
Apr 16 16:31:12.384731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384690    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="main"
Apr 16 16:31:12.384731 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384696    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="main"
Apr 16 16:31:12.384935 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384749    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="main"
Apr 16 16:31:12.384935 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.384758    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="4e857ccc-440e-4b23-9ff7-93cd5e1f0e88" containerName="tokenizer"
Apr 16 16:31:12.388293 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.388260    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.391477 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.391432    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-h2274\""
Apr 16 16:31:12.391673 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.391662    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Apr 16 16:31:12.404572 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.404550    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"]
Apr 16 16:31:12.529051 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.529024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8648d8-54b3-43a8-988f-17fdd4599fa3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.529190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.529063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.529190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.529125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.529190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.529162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.529190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.529190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.529329 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.529208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6s4x\" (UniqueName: \"kubernetes.io/projected/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kube-api-access-w6s4x\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630368 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630338    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630518 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630375    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630518 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630400    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630518 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630430    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w6s4x\" (UniqueName: \"kubernetes.io/projected/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kube-api-access-w6s4x\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630518 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630507    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8648d8-54b3-43a8-988f-17fdd4599fa3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630739 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630557    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630806 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630747    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630863 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630816    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.630914 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.630888    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.632765 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.632745    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.633014 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.632996    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8648d8-54b3-43a8-988f-17fdd4599fa3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.646998 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.646942    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6s4x\" (UniqueName: \"kubernetes.io/projected/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kube-api-access-w6s4x\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.659569 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.659545    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"]
Apr 16 16:31:12.663881 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.663857    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.668734 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.668709    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-epp-sa-dockercfg-bvqwd\""
Apr 16 16:31:12.678417 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.678393    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"]
Apr 16 16:31:12.700265 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.700245    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:12.832153 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.832118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2d7m\" (UniqueName: \"kubernetes.io/projected/2e3d1646-cf98-4075-a499-684101338b20-kube-api-access-x2d7m\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.832300 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.832173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.832300 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.832277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.832438 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.832315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.832438 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.832339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3d1646-cf98-4075-a499-684101338b20-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.832438 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.832374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.839281 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.839252    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"]
Apr 16 16:31:12.841438 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:31:12.841371    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8648d8_54b3_43a8_988f_17fdd4599fa3.slice/crio-dd0a92db8c0cb64b5344d6aab2540e00574eaa7ed1ccffb92e5f9f252b851457 WatchSource:0}: Error finding container dd0a92db8c0cb64b5344d6aab2540e00574eaa7ed1ccffb92e5f9f252b851457: Status 404 returned error can't find the container with id dd0a92db8c0cb64b5344d6aab2540e00574eaa7ed1ccffb92e5f9f252b851457
Apr 16 16:31:12.933756 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.933694    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.933756 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.933743    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.933897 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.933867    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.933943 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.933902    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3d1646-cf98-4075-a499-684101338b20-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.933943 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.933923    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.934015 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.933981    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x2d7m\" (UniqueName: \"kubernetes.io/projected/2e3d1646-cf98-4075-a499-684101338b20-kube-api-access-x2d7m\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.934062 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.934043    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-tmp\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.934119 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.934092    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-uds\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.934221 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.934194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-cache\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.934326 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.934292    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-kserve-provision-location\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.936374 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.936353    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3d1646-cf98-4075-a499-684101338b20-tls-certs\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.942558 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.942541    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2d7m\" (UniqueName: \"kubernetes.io/projected/2e3d1646-cf98-4075-a499-684101338b20-kube-api-access-x2d7m\") pod \"llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") " pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:12.975440 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:12.975416    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:13.101741 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:13.101694    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"]
Apr 16 16:31:13.102869 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:31:13.102842    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3d1646_cf98_4075_a499_684101338b20.slice/crio-65de6dc7711e9ab1b7cfbb6932a473dda7fa54efdf2e5d0fd52b51dd04f04018 WatchSource:0}: Error finding container 65de6dc7711e9ab1b7cfbb6932a473dda7fa54efdf2e5d0fd52b51dd04f04018: Status 404 returned error can't find the container with id 65de6dc7711e9ab1b7cfbb6932a473dda7fa54efdf2e5d0fd52b51dd04f04018
Apr 16 16:31:13.104887 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:13.104867    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:31:13.209388 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:13.209359    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerStarted","Data":"5fea8656919f624577c4ecb628e3bc29b2ee6ecedf3f2940644f5901940d461a"}
Apr 16 16:31:13.209575 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:13.209393    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerStarted","Data":"65de6dc7711e9ab1b7cfbb6932a473dda7fa54efdf2e5d0fd52b51dd04f04018"}
Apr 16 16:31:13.210572 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:13.210531    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerStarted","Data":"dd0a92db8c0cb64b5344d6aab2540e00574eaa7ed1ccffb92e5f9f252b851457"}
Apr 16 16:31:14.215583 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:14.215552    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerStarted","Data":"108634c8059c54d35708f99fa2a500288d9c40f27bd519f86a6d30481a0e208b"}
Apr 16 16:31:14.215987 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:14.215683    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:14.216921 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:14.216899    2578 generic.go:358] "Generic (PLEG): container finished" podID="2e3d1646-cf98-4075-a499-684101338b20" containerID="5fea8656919f624577c4ecb628e3bc29b2ee6ecedf3f2940644f5901940d461a" exitCode=0
Apr 16 16:31:14.217020 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:14.217007    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerDied","Data":"5fea8656919f624577c4ecb628e3bc29b2ee6ecedf3f2940644f5901940d461a"}
Apr 16 16:31:15.223236 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:15.223193    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerStarted","Data":"31f9e488f7a85b4ecc8d784a5139b609532278893b7e84f2e89e5eee53736bf6"}
Apr 16 16:31:15.225819 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:15.225784    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerStarted","Data":"15d5be4921736856efea4064cce2c5328f3fa8269cf2fffdd97c9d64a393d263"}
Apr 16 16:31:15.225970 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:15.225827    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerStarted","Data":"d85889ec4b3e6a913b146b2c2cc0be2c33f9a14c60b719e5266660d86be5895a"}
Apr 16 16:31:15.225970 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:15.225892    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:15.263097 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:15.263044    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" podStartSLOduration=3.263019055 podStartE2EDuration="3.263019055s" podCreationTimestamp="2026-04-16 16:31:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:31:15.261883929 +0000 UTC m=+1796.657968346" watchObservedRunningTime="2026-04-16 16:31:15.263019055 +0000 UTC m=+1796.659103500"
Apr 16 16:31:19.197565 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:19.197540    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:31:19.202142 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:19.202120    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:31:19.253536 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:19.253500    2578 generic.go:358] "Generic (PLEG): container finished" podID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerID="31f9e488f7a85b4ecc8d784a5139b609532278893b7e84f2e89e5eee53736bf6" exitCode=0
Apr 16 16:31:19.253689 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:19.253574    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerDied","Data":"31f9e488f7a85b4ecc8d784a5139b609532278893b7e84f2e89e5eee53736bf6"}
Apr 16 16:31:20.261723 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:20.261678    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerStarted","Data":"cd0ed46c135cf80bd286f242208299a4650aceb5f226d0bc405d0b1ddac8af39"}
Apr 16 16:31:20.284310 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:20.284255    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podStartSLOduration=7.122336488 podStartE2EDuration="8.284240204s" podCreationTimestamp="2026-04-16 16:31:12 +0000 UTC" firstStartedPulling="2026-04-16 16:31:12.8454308 +0000 UTC m=+1794.241515216" lastFinishedPulling="2026-04-16 16:31:14.00733453 +0000 UTC m=+1795.403418932" observedRunningTime="2026-04-16 16:31:20.281816473 +0000 UTC m=+1801.677900893" watchObservedRunningTime="2026-04-16 16:31:20.284240204 +0000 UTC m=+1801.680324631"
Apr 16 16:31:22.700995 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:22.700963    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:22.701408 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:22.701034    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:22.702365 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:22.702338    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:31:22.975695 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:22.975662    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:22.975856 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:22.975717    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:22.978769 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:22.978747    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:23.274555 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:23.274438    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:32.700918 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:32.700869    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:31:33.149271 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:33.149237    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"]
Apr 16 16:31:33.149624 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:33.149595    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main" containerID="cri-o://a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6" gracePeriod=30
Apr 16 16:31:33.290595 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:33.290569    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:31:42.701117 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:42.701056    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:31:45.283658 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:45.283625    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:31:52.700966 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:31:52.700913    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:32:02.700644 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:02.700602    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:32:03.382096 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.382069    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:32:03.452559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.452482    2578 generic.go:358] "Generic (PLEG): container finished" podID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerID="a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6" exitCode=137
Apr 16 16:32:03.452559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.452549    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" event={"ID":"c5356d2f-1700-48f6-beaf-f5bc3957d89d","Type":"ContainerDied","Data":"a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6"}
Apr 16 16:32:03.452559 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.452557    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"
Apr 16 16:32:03.452772 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.452581    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g" event={"ID":"c5356d2f-1700-48f6-beaf-f5bc3957d89d","Type":"ContainerDied","Data":"4441ec582e8916732d3e72f3e3e9a0d2fa8892034326510e3875c7391f0868ec"}
Apr 16 16:32:03.452772 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.452600    2578 scope.go:117] "RemoveContainer" containerID="a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6"
Apr 16 16:32:03.478663 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.478637    2578 scope.go:117] "RemoveContainer" containerID="e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568"
Apr 16 16:32:03.480907 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.480885    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5356d2f-1700-48f6-beaf-f5bc3957d89d-tls-certs\") pod \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") "
Apr 16 16:32:03.481013 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.480948    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kserve-provision-location\") pod \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") "
Apr 16 16:32:03.481013 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.480984    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-model-cache\") pod \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") "
Apr 16 16:32:03.481130 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.481022    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-dshm\") pod \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") "
Apr 16 16:32:03.481130 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.481068    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-home\") pod \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") "
Apr 16 16:32:03.481130 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.481115    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crq42\" (UniqueName: \"kubernetes.io/projected/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kube-api-access-crq42\") pod \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\" (UID: \"c5356d2f-1700-48f6-beaf-f5bc3957d89d\") "
Apr 16 16:32:03.481296 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.481261    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5356d2f-1700-48f6-beaf-f5bc3957d89d" (UID: "c5356d2f-1700-48f6-beaf-f5bc3957d89d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:32:03.481523 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.481493    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:32:03.481635 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.481541    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-home" (OuterVolumeSpecName: "home") pod "c5356d2f-1700-48f6-beaf-f5bc3957d89d" (UID: "c5356d2f-1700-48f6-beaf-f5bc3957d89d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:32:03.483412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.483389    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-dshm" (OuterVolumeSpecName: "dshm") pod "c5356d2f-1700-48f6-beaf-f5bc3957d89d" (UID: "c5356d2f-1700-48f6-beaf-f5bc3957d89d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:32:03.483536 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.483433    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5356d2f-1700-48f6-beaf-f5bc3957d89d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5356d2f-1700-48f6-beaf-f5bc3957d89d" (UID: "c5356d2f-1700-48f6-beaf-f5bc3957d89d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:32:03.483684 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.483666    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kube-api-access-crq42" (OuterVolumeSpecName: "kube-api-access-crq42") pod "c5356d2f-1700-48f6-beaf-f5bc3957d89d" (UID: "c5356d2f-1700-48f6-beaf-f5bc3957d89d"). InnerVolumeSpecName "kube-api-access-crq42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:32:03.545181 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.545143    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5356d2f-1700-48f6-beaf-f5bc3957d89d" (UID: "c5356d2f-1700-48f6-beaf-f5bc3957d89d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:32:03.562259 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.562237    2578 scope.go:117] "RemoveContainer" containerID="a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6"
Apr 16 16:32:03.562631 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:32:03.562601    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6\": container with ID starting with a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6 not found: ID does not exist" containerID="a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6"
Apr 16 16:32:03.562746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.562641    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6"} err="failed to get container status \"a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6\": rpc error: code = NotFound desc = could not find container \"a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6\": container with ID starting with a402ab37fcf4e5e531bf542a391976cf36a41f9416fc8670ae7d9ef8b3d69bc6 not found: ID does not exist"
Apr 16 16:32:03.562746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.562661    2578 scope.go:117] "RemoveContainer" containerID="e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568"
Apr 16 16:32:03.562972 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:32:03.562947    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568\": container with ID starting with e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568 not found: ID does not exist" containerID="e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568"
Apr 16 16:32:03.563033 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.562980    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568"} err="failed to get container status \"e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568\": rpc error: code = NotFound desc = could not find container \"e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568\": container with ID starting with e745499d08703ddf23b6afffc1ba3eeae03dfc7b6ded49cd6f4d4493a84e2568 not found: ID does not exist"
Apr 16 16:32:03.582515 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.582481    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:32:03.582515 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.582511    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:32:03.582689 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.582523    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5356d2f-1700-48f6-beaf-f5bc3957d89d-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:32:03.582689 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.582554    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crq42\" (UniqueName: \"kubernetes.io/projected/c5356d2f-1700-48f6-beaf-f5bc3957d89d-kube-api-access-crq42\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:32:03.582689 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.582565    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5356d2f-1700-48f6-beaf-f5bc3957d89d-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:32:03.775701 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.775628    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"]
Apr 16 16:32:03.778473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:03.778434    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-64dfc65446-25j7g"]
Apr 16 16:32:05.108857 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:05.108823    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" path="/var/lib/kubelet/pods/c5356d2f-1700-48f6-beaf-f5bc3957d89d/volumes"
Apr 16 16:32:12.701470 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:12.701401    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:32:22.701025 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:22.700977    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:32:32.701144 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:32.701034    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:32:42.701351 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:42.701305    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8001/health\": dial tcp 10.134.0.60:8001: connect: connection refused"
Apr 16 16:32:52.710028 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:52.709996    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:32:52.721966 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:32:52.721945    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:33:15.691280 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:15.691248    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"]
Apr 16 16:33:15.691953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:15.691676    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="main" containerID="cri-o://d85889ec4b3e6a913b146b2c2cc0be2c33f9a14c60b719e5266660d86be5895a" gracePeriod=30
Apr 16 16:33:15.691953 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:15.691837    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="tokenizer" containerID="cri-o://15d5be4921736856efea4064cce2c5328f3fa8269cf2fffdd97c9d64a393d263" gracePeriod=30
Apr 16 16:33:15.696065 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:15.695667    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"]
Apr 16 16:33:15.696730 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:15.696678    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main" containerID="cri-o://cd0ed46c135cf80bd286f242208299a4650aceb5f226d0bc405d0b1ddac8af39" gracePeriod=30
Apr 16 16:33:16.753105 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.753071    2578 generic.go:358] "Generic (PLEG): container finished" podID="2e3d1646-cf98-4075-a499-684101338b20" containerID="15d5be4921736856efea4064cce2c5328f3fa8269cf2fffdd97c9d64a393d263" exitCode=0
Apr 16 16:33:16.753105 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.753099    2578 generic.go:358] "Generic (PLEG): container finished" podID="2e3d1646-cf98-4075-a499-684101338b20" containerID="d85889ec4b3e6a913b146b2c2cc0be2c33f9a14c60b719e5266660d86be5895a" exitCode=0
Apr 16 16:33:16.753500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.753150    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerDied","Data":"15d5be4921736856efea4064cce2c5328f3fa8269cf2fffdd97c9d64a393d263"}
Apr 16 16:33:16.753500 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.753189    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerDied","Data":"d85889ec4b3e6a913b146b2c2cc0be2c33f9a14c60b719e5266660d86be5895a"}
Apr 16 16:33:16.848266 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.848244    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:33:16.894975 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.894914    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3d1646-cf98-4075-a499-684101338b20-tls-certs\") pod \"2e3d1646-cf98-4075-a499-684101338b20\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") "
Apr 16 16:33:16.895082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895002    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-cache\") pod \"2e3d1646-cf98-4075-a499-684101338b20\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") "
Apr 16 16:33:16.895082 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895063    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2d7m\" (UniqueName: \"kubernetes.io/projected/2e3d1646-cf98-4075-a499-684101338b20-kube-api-access-x2d7m\") pod \"2e3d1646-cf98-4075-a499-684101338b20\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") "
Apr 16 16:33:16.895151 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895099    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-uds\") pod \"2e3d1646-cf98-4075-a499-684101338b20\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") "
Apr 16 16:33:16.895151 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895130    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "2e3d1646-cf98-4075-a499-684101338b20" (UID: "2e3d1646-cf98-4075-a499-684101338b20"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:16.895151 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895143    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-tmp\") pod \"2e3d1646-cf98-4075-a499-684101338b20\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") "
Apr 16 16:33:16.895247 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895174    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-kserve-provision-location\") pod \"2e3d1646-cf98-4075-a499-684101338b20\" (UID: \"2e3d1646-cf98-4075-a499-684101338b20\") "
Apr 16 16:33:16.895344 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895294    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2e3d1646-cf98-4075-a499-684101338b20" (UID: "2e3d1646-cf98-4075-a499-684101338b20"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:16.895550 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895529    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:16.895550 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895534    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "2e3d1646-cf98-4075-a499-684101338b20" (UID: "2e3d1646-cf98-4075-a499-684101338b20"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:16.895550 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895552    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:16.895956 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.895897    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e3d1646-cf98-4075-a499-684101338b20" (UID: "2e3d1646-cf98-4075-a499-684101338b20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:16.897140 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.897120    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3d1646-cf98-4075-a499-684101338b20-kube-api-access-x2d7m" (OuterVolumeSpecName: "kube-api-access-x2d7m") pod "2e3d1646-cf98-4075-a499-684101338b20" (UID: "2e3d1646-cf98-4075-a499-684101338b20"). InnerVolumeSpecName "kube-api-access-x2d7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:33:16.897232 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.897216    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3d1646-cf98-4075-a499-684101338b20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e3d1646-cf98-4075-a499-684101338b20" (UID: "2e3d1646-cf98-4075-a499-684101338b20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:33:16.996110 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.996080    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e3d1646-cf98-4075-a499-684101338b20-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:16.996110 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.996109    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2d7m\" (UniqueName: \"kubernetes.io/projected/2e3d1646-cf98-4075-a499-684101338b20-kube-api-access-x2d7m\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:16.996110 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.996118    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:16.996271 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:16.996127    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e3d1646-cf98-4075-a499-684101338b20-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:17.758367 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:17.758330    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9" event={"ID":"2e3d1646-cf98-4075-a499-684101338b20","Type":"ContainerDied","Data":"65de6dc7711e9ab1b7cfbb6932a473dda7fa54efdf2e5d0fd52b51dd04f04018"}
Apr 16 16:33:17.758367 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:17.758366    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"
Apr 16 16:33:17.758810 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:17.758382    2578 scope.go:117] "RemoveContainer" containerID="15d5be4921736856efea4064cce2c5328f3fa8269cf2fffdd97c9d64a393d263"
Apr 16 16:33:17.769355 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:17.769329    2578 scope.go:117] "RemoveContainer" containerID="d85889ec4b3e6a913b146b2c2cc0be2c33f9a14c60b719e5266660d86be5895a"
Apr 16 16:33:17.777480 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:17.777435    2578 scope.go:117] "RemoveContainer" containerID="5fea8656919f624577c4ecb628e3bc29b2ee6ecedf3f2940644f5901940d461a"
Apr 16 16:33:17.784572 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:17.782412    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"]
Apr 16 16:33:17.788955 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:17.788931    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8f1a6f044e8c7a4d31a250e0c4861caf-kserve-router-scherlds9"]
Apr 16 16:33:19.108093 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:19.108055    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e3d1646-cf98-4075-a499-684101338b20" path="/var/lib/kubelet/pods/2e3d1646-cf98-4075-a499-684101338b20/volumes"
Apr 16 16:33:21.606975 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.606941    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"]
Apr 16 16:33:21.607395 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607355    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="main"
Apr 16 16:33:21.607395 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607368    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="main"
Apr 16 16:33:21.607395 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607379    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main"
Apr 16 16:33:21.607395 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607385    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607399    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="tokenizer"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607405    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="tokenizer"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607426    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="storage-initializer"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607436    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="storage-initializer"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607459    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="storage-initializer"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607468    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="storage-initializer"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607545    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="c5356d2f-1700-48f6-beaf-f5bc3957d89d" containerName="main"
Apr 16 16:33:21.607554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607557    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="main"
Apr 16 16:33:21.607818 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.607563    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="2e3d1646-cf98-4075-a499-684101338b20" containerName="tokenizer"
Apr 16 16:33:21.612566 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.612544    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.615035 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.615012    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Apr 16 16:33:21.615215 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.615198    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-sknxj\""
Apr 16 16:33:21.622151 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.622128    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"]
Apr 16 16:33:21.729866 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.729834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-dshm\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.730023 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.729871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvc2v\" (UniqueName: \"kubernetes.io/projected/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kube-api-access-xvc2v\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.730023 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.729906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-home\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.730023 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.729946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-model-cache\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.730023 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.729986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.730201 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.730046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fee1b-f258-4ef6-989c-8bbb284fce53-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831027 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831000    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-dshm\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831195 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831042    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xvc2v\" (UniqueName: \"kubernetes.io/projected/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kube-api-access-xvc2v\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831195 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831085    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-home\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831195 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831113    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-model-cache\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831195 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831159    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831425 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831230    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fee1b-f258-4ef6-989c-8bbb284fce53-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831749 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831728    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-model-cache\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831857 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831793    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-home\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.831857 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.831817    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.833772 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.833752    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fee1b-f258-4ef6-989c-8bbb284fce53-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.833854 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.833814    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-dshm\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.849661 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.849639    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvc2v\" (UniqueName: \"kubernetes.io/projected/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kube-api-access-xvc2v\") pod \"custom-route-timeout-pd-test-kserve-996756845-nvhq9\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:21.922523 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:21.922456    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:22.062087 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:22.061634    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"]
Apr 16 16:33:22.066696 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:33:22.066664    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1fee1b_f258_4ef6_989c_8bbb284fce53.slice/crio-68c63c9463d1b17ad76c753299e4a596c5cf858561ae4ac4af25f5a7cf82e7f6 WatchSource:0}: Error finding container 68c63c9463d1b17ad76c753299e4a596c5cf858561ae4ac4af25f5a7cf82e7f6: Status 404 returned error can't find the container with id 68c63c9463d1b17ad76c753299e4a596c5cf858561ae4ac4af25f5a7cf82e7f6
Apr 16 16:33:22.780305 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:22.780272    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerStarted","Data":"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"}
Apr 16 16:33:22.780305 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:22.780306    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerStarted","Data":"68c63c9463d1b17ad76c753299e4a596c5cf858561ae4ac4af25f5a7cf82e7f6"}
Apr 16 16:33:22.780844 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:22.780442    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:23.787641 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:23.787604    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerStarted","Data":"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"}
Apr 16 16:33:26.802989 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:26.802962    2578 generic.go:358] "Generic (PLEG): container finished" podID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerID="825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe" exitCode=0
Apr 16 16:33:26.803362 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:26.803041    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerDied","Data":"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"}
Apr 16 16:33:27.808756 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:27.808724    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerStarted","Data":"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"}
Apr 16 16:33:27.831419 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:27.831370    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podStartSLOduration=6.831357061 podStartE2EDuration="6.831357061s" podCreationTimestamp="2026-04-16 16:33:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:33:27.828790109 +0000 UTC m=+1929.224874527" watchObservedRunningTime="2026-04-16 16:33:27.831357061 +0000 UTC m=+1929.227441476"
Apr 16 16:33:31.923647 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:31.923614    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:31.924091 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:31.923668    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:31.924872 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:31.924845    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:33:41.924089 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:41.924031    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:33:41.945586 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:41.945562    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:33:45.697670 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.697620    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="llm-d-routing-sidecar" containerID="cri-o://108634c8059c54d35708f99fa2a500288d9c40f27bd519f86a6d30481a0e208b" gracePeriod=2
Apr 16 16:33:45.896310 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.896275    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7_ed8648d8-54b3-43a8-988f-17fdd4599fa3/main/0.log"
Apr 16 16:33:45.897053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.897023    2578 generic.go:358] "Generic (PLEG): container finished" podID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerID="cd0ed46c135cf80bd286f242208299a4650aceb5f226d0bc405d0b1ddac8af39" exitCode=137
Apr 16 16:33:45.897053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.897047    2578 generic.go:358] "Generic (PLEG): container finished" podID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerID="108634c8059c54d35708f99fa2a500288d9c40f27bd519f86a6d30481a0e208b" exitCode=0
Apr 16 16:33:45.897309 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.897138    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerDied","Data":"cd0ed46c135cf80bd286f242208299a4650aceb5f226d0bc405d0b1ddac8af39"}
Apr 16 16:33:45.897309 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.897165    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerDied","Data":"108634c8059c54d35708f99fa2a500288d9c40f27bd519f86a6d30481a0e208b"}
Apr 16 16:33:45.949876 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.949818    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7_ed8648d8-54b3-43a8-988f-17fdd4599fa3/main/0.log"
Apr 16 16:33:45.950479 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:45.950425    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:33:46.047175 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047140    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8648d8-54b3-43a8-988f-17fdd4599fa3-tls-certs\") pod \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") "
Apr 16 16:33:46.047363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047214    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6s4x\" (UniqueName: \"kubernetes.io/projected/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kube-api-access-w6s4x\") pod \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") "
Apr 16 16:33:46.047363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047273    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-home\") pod \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") "
Apr 16 16:33:46.047363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047303    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kserve-provision-location\") pod \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") "
Apr 16 16:33:46.047363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047326    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-model-cache\") pod \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") "
Apr 16 16:33:46.047363 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047361    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-dshm\") pod \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\" (UID: \"ed8648d8-54b3-43a8-988f-17fdd4599fa3\") "
Apr 16 16:33:46.047664 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047624    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed8648d8-54b3-43a8-988f-17fdd4599fa3" (UID: "ed8648d8-54b3-43a8-988f-17fdd4599fa3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:46.047793 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.047775    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-home" (OuterVolumeSpecName: "home") pod "ed8648d8-54b3-43a8-988f-17fdd4599fa3" (UID: "ed8648d8-54b3-43a8-988f-17fdd4599fa3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:46.049705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.049668    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kube-api-access-w6s4x" (OuterVolumeSpecName: "kube-api-access-w6s4x") pod "ed8648d8-54b3-43a8-988f-17fdd4599fa3" (UID: "ed8648d8-54b3-43a8-988f-17fdd4599fa3"). InnerVolumeSpecName "kube-api-access-w6s4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:33:46.049843 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.049734    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-dshm" (OuterVolumeSpecName: "dshm") pod "ed8648d8-54b3-43a8-988f-17fdd4599fa3" (UID: "ed8648d8-54b3-43a8-988f-17fdd4599fa3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:46.049931 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.049843    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8648d8-54b3-43a8-988f-17fdd4599fa3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed8648d8-54b3-43a8-988f-17fdd4599fa3" (UID: "ed8648d8-54b3-43a8-988f-17fdd4599fa3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:33:46.104109 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.104059    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed8648d8-54b3-43a8-988f-17fdd4599fa3" (UID: "ed8648d8-54b3-43a8-988f-17fdd4599fa3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:33:46.148756 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.148711    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8648d8-54b3-43a8-988f-17fdd4599fa3-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:46.148756 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.148759    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6s4x\" (UniqueName: \"kubernetes.io/projected/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kube-api-access-w6s4x\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:46.148951 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.148775    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:46.148951 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.148791    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:46.148951 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.148804    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:46.148951 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.148817    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed8648d8-54b3-43a8-988f-17fdd4599fa3-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:33:46.902169 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.902144    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7_ed8648d8-54b3-43a8-988f-17fdd4599fa3/main/0.log"
Apr 16 16:33:46.902895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.902865    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7" event={"ID":"ed8648d8-54b3-43a8-988f-17fdd4599fa3","Type":"ContainerDied","Data":"dd0a92db8c0cb64b5344d6aab2540e00574eaa7ed1ccffb92e5f9f252b851457"}
Apr 16 16:33:46.903031 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.902914    2578 scope.go:117] "RemoveContainer" containerID="cd0ed46c135cf80bd286f242208299a4650aceb5f226d0bc405d0b1ddac8af39"
Apr 16 16:33:46.903031 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.902917    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"
Apr 16 16:33:46.926058 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.926035    2578 scope.go:117] "RemoveContainer" containerID="31f9e488f7a85b4ecc8d784a5139b609532278893b7e84f2e89e5eee53736bf6"
Apr 16 16:33:46.927517 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.927494    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"]
Apr 16 16:33:46.932198 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.932175    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-779d88d-99zk7"]
Apr 16 16:33:46.989609 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:46.989590    2578 scope.go:117] "RemoveContainer" containerID="108634c8059c54d35708f99fa2a500288d9c40f27bd519f86a6d30481a0e208b"
Apr 16 16:33:47.108133 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:47.108101    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" path="/var/lib/kubelet/pods/ed8648d8-54b3-43a8-988f-17fdd4599fa3/volumes"
Apr 16 16:33:51.923584 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:33:51.923523    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:34:00.889730 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.889641    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890201    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main"
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890217    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main"
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890237    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="llm-d-routing-sidecar"
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890242    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="llm-d-routing-sidecar"
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890258    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="storage-initializer"
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890264    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="storage-initializer"
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890326    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="main"
Apr 16 16:34:00.892531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.890338    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ed8648d8-54b3-43a8-988f-17fdd4599fa3" containerName="llm-d-routing-sidecar"
Apr 16 16:34:00.893381 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.893362    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:00.896580 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.896549    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Apr 16 16:34:00.897398 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.897377    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-7c864\""
Apr 16 16:34:00.910785 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.910763    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 16 16:34:00.967761 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.967735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjspz\" (UniqueName: \"kubernetes.io/projected/7b5d646f-01a1-42d1-87b2-33f760b01385-kube-api-access-pjspz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:00.967903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.967778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5d646f-01a1-42d1-87b2-33f760b01385-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:00.967903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.967880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:00.967981 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.967922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:00.968018 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.967979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:00.968018 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:00.968015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.063672 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.063637    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"]
Apr 16 16:34:01.067527 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.067509    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.068790 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.068769    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.068903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.068804    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.068903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.068848    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pjspz\" (UniqueName: \"kubernetes.io/projected/7b5d646f-01a1-42d1-87b2-33f760b01385-kube-api-access-pjspz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.068903 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.068896    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5d646f-01a1-42d1-87b2-33f760b01385-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.069084 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.068967    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.069084 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.068993    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.069274 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.069225    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.069364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.069266    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.069364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.069338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.070526 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.070303    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5ec-epp-sa-dockercfg-hdhp2\""
Apr 16 16:34:01.071902 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.071677    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.072157 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.072126    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5d646f-01a1-42d1-87b2-33f760b01385-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.077537 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.077514    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"]
Apr 16 16:34:01.078103 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.078079    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjspz\" (UniqueName: \"kubernetes.io/projected/7b5d646f-01a1-42d1-87b2-33f760b01385-kube-api-access-pjspz\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.170408 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.170329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.170408 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.170366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xlv\" (UniqueName: \"kubernetes.io/projected/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kube-api-access-79xlv\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.170642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.170503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.170642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.170569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.170642 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.170627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.170774 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.170747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.204440 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.204416    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:01.271391 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.271558 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271400    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-79xlv\" (UniqueName: \"kubernetes.io/projected/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kube-api-access-79xlv\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.271558 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271436    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.271558 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271490    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.271558 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271542    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.271698 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271602    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.272057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271767    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kserve-provision-location\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.272057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271902    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-cache\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.272057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.271955    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-tmp\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.272057 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.272031    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-uds\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.274303 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.274225    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tls-certs\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.281949 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.281907    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xlv\" (UniqueName: \"kubernetes.io/projected/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kube-api-access-79xlv\") pod \"llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") " pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.338300 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.338271    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 16 16:34:01.339919 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:34:01.339893    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5d646f_01a1_42d1_87b2_33f760b01385.slice/crio-cc2ca2c74f0b4aa9f9b8e5217c4be2fbdbcc6098759394add2f9d4fcb6412f62 WatchSource:0}: Error finding container cc2ca2c74f0b4aa9f9b8e5217c4be2fbdbcc6098759394add2f9d4fcb6412f62: Status 404 returned error can't find the container with id cc2ca2c74f0b4aa9f9b8e5217c4be2fbdbcc6098759394add2f9d4fcb6412f62
Apr 16 16:34:01.399372 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.399346    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:01.533594 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.533562    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"]
Apr 16 16:34:01.533683 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:34:01.533624    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d7bf3d_6677_4d4b_8ce5_6c0310fe8522.slice/crio-7a7304cdb1debff0f29ea13d399bf150bf145d32185b9108e2fe3244e52de2a3 WatchSource:0}: Error finding container 7a7304cdb1debff0f29ea13d399bf150bf145d32185b9108e2fe3244e52de2a3: Status 404 returned error can't find the container with id 7a7304cdb1debff0f29ea13d399bf150bf145d32185b9108e2fe3244e52de2a3
Apr 16 16:34:01.923572 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.923523    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:34:01.966717 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.966677    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerStarted","Data":"ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21"}
Apr 16 16:34:01.966717 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.966710    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerStarted","Data":"7a7304cdb1debff0f29ea13d399bf150bf145d32185b9108e2fe3244e52de2a3"}
Apr 16 16:34:01.968232 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.968195    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"7b5d646f-01a1-42d1-87b2-33f760b01385","Type":"ContainerStarted","Data":"55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e"}
Apr 16 16:34:01.968232 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:01.968228    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"7b5d646f-01a1-42d1-87b2-33f760b01385","Type":"ContainerStarted","Data":"cc2ca2c74f0b4aa9f9b8e5217c4be2fbdbcc6098759394add2f9d4fcb6412f62"}
Apr 16 16:34:02.974520 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:02.974481    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerDied","Data":"ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21"}
Apr 16 16:34:02.974520 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:02.974471    2578 generic.go:358] "Generic (PLEG): container finished" podID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerID="ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21" exitCode=0
Apr 16 16:34:03.983999 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:03.983952    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerStarted","Data":"831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6"}
Apr 16 16:34:03.983999 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:03.984004    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerStarted","Data":"7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3"}
Apr 16 16:34:03.984531 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:03.984089    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:04.009307 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:04.009192    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" podStartSLOduration=3.009123542 podStartE2EDuration="3.009123542s" podCreationTimestamp="2026-04-16 16:34:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:34:04.00773278 +0000 UTC m=+1965.403817201" watchObservedRunningTime="2026-04-16 16:34:04.009123542 +0000 UTC m=+1965.405207956"
Apr 16 16:34:05.995226 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:05.995194    2578 generic.go:358] "Generic (PLEG): container finished" podID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerID="55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e" exitCode=0
Apr 16 16:34:05.995746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:05.995272    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"7b5d646f-01a1-42d1-87b2-33f760b01385","Type":"ContainerDied","Data":"55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e"}
Apr 16 16:34:07.002431 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:07.002394    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"7b5d646f-01a1-42d1-87b2-33f760b01385","Type":"ContainerStarted","Data":"d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4"}
Apr 16 16:34:07.025860 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:07.025802    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=7.02578194 podStartE2EDuration="7.02578194s" podCreationTimestamp="2026-04-16 16:34:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:34:07.023014023 +0000 UTC m=+1968.419098445" watchObservedRunningTime="2026-04-16 16:34:07.02578194 +0000 UTC m=+1968.421866365"
Apr 16 16:34:11.204926 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:11.204895    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:11.206210 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:11.206179    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:34:11.399758 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:11.399718    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:11.399758 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:11.399765    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:11.402927 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:11.402900    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:11.923965 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:11.923917    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:34:12.026636 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:12.026604    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:21.205644 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:21.205598    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:34:21.923574 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:21.923536    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:34:31.205159 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:31.205120    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:34:31.205592 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:31.205375    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:34:31.923705 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:31.923646    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:34:33.030655 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:33.030627    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:34:41.205856 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:41.205821    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:34:41.923789 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:41.923741    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:34:51.205412 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:51.205358    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:34:51.923493 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:34:51.923424    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Apr 16 16:35:01.205316 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:01.205260    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:35:01.932948 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:01.932914    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:35:01.950269 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:01.950244    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:35:11.205812 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:11.205774    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:35:19.201800 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:19.201767    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"]
Apr 16 16:35:19.202176 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:19.202129    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main" containerID="cri-o://d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845" gracePeriod=30
Apr 16 16:35:21.204985 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:21.204939    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:35:31.205800 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:31.205713    2578 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Apr 16 16:35:41.213900 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:41.213864    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:35:41.221317 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:41.221294    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:35:49.202942 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.202900    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="llm-d-routing-sidecar" containerID="cri-o://bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274" gracePeriod=2
Apr 16 16:35:49.445784 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.445760    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-996756845-nvhq9_7d1fee1b-f258-4ef6-989c-8bbb284fce53/main/0.log"
Apr 16 16:35:49.446474 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.446433    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:35:49.475891 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.475865    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-996756845-nvhq9_7d1fee1b-f258-4ef6-989c-8bbb284fce53/main/0.log"
Apr 16 16:35:49.476643 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.476610    2578 generic.go:358] "Generic (PLEG): container finished" podID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerID="d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845" exitCode=137
Apr 16 16:35:49.476643 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.476642    2578 generic.go:358] "Generic (PLEG): container finished" podID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerID="bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274" exitCode=0
Apr 16 16:35:49.476836 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.476688    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerDied","Data":"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"}
Apr 16 16:35:49.476836 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.476738    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerDied","Data":"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"}
Apr 16 16:35:49.476836 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.476747    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"
Apr 16 16:35:49.476836 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.476765    2578 scope.go:117] "RemoveContainer" containerID="d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"
Apr 16 16:35:49.477009 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.476753    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9" event={"ID":"7d1fee1b-f258-4ef6-989c-8bbb284fce53","Type":"ContainerDied","Data":"68c63c9463d1b17ad76c753299e4a596c5cf858561ae4ac4af25f5a7cf82e7f6"}
Apr 16 16:35:49.496806 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.496779    2578 scope.go:117] "RemoveContainer" containerID="825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"
Apr 16 16:35:49.553995 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.553970    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-home\") pod \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") "
Apr 16 16:35:49.554153 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554030    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-model-cache\") pod \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") "
Apr 16 16:35:49.554153 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554070    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fee1b-f258-4ef6-989c-8bbb284fce53-tls-certs\") pod \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") "
Apr 16 16:35:49.554153 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554096    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvc2v\" (UniqueName: \"kubernetes.io/projected/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kube-api-access-xvc2v\") pod \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") "
Apr 16 16:35:49.554153 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554141    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-dshm\") pod \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") "
Apr 16 16:35:49.554366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554171    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kserve-provision-location\") pod \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\" (UID: \"7d1fee1b-f258-4ef6-989c-8bbb284fce53\") "
Apr 16 16:35:49.554366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554308    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d1fee1b-f258-4ef6-989c-8bbb284fce53" (UID: "7d1fee1b-f258-4ef6-989c-8bbb284fce53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:35:49.554366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554346    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-home" (OuterVolumeSpecName: "home") pod "7d1fee1b-f258-4ef6-989c-8bbb284fce53" (UID: "7d1fee1b-f258-4ef6-989c-8bbb284fce53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:35:49.554669 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554643    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:35:49.554786 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.554679    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:35:49.556419 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.556367    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d1fee1b-f258-4ef6-989c-8bbb284fce53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d1fee1b-f258-4ef6-989c-8bbb284fce53" (UID: "7d1fee1b-f258-4ef6-989c-8bbb284fce53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:35:49.556540 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.556490    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-dshm" (OuterVolumeSpecName: "dshm") pod "7d1fee1b-f258-4ef6-989c-8bbb284fce53" (UID: "7d1fee1b-f258-4ef6-989c-8bbb284fce53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:35:49.556708 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.556682    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kube-api-access-xvc2v" (OuterVolumeSpecName: "kube-api-access-xvc2v") pod "7d1fee1b-f258-4ef6-989c-8bbb284fce53" (UID: "7d1fee1b-f258-4ef6-989c-8bbb284fce53"). InnerVolumeSpecName "kube-api-access-xvc2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:35:49.565312 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.565123    2578 scope.go:117] "RemoveContainer" containerID="bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"
Apr 16 16:35:49.572923 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.572903    2578 scope.go:117] "RemoveContainer" containerID="d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"
Apr 16 16:35:49.573161 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:35:49.573142    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845\": container with ID starting with d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845 not found: ID does not exist" containerID="d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"
Apr 16 16:35:49.573222 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573169    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"} err="failed to get container status \"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845\": rpc error: code = NotFound desc = could not find container \"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845\": container with ID starting with d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845 not found: ID does not exist"
Apr 16 16:35:49.573222 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573188    2578 scope.go:117] "RemoveContainer" containerID="825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"
Apr 16 16:35:49.573426 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:35:49.573408    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe\": container with ID starting with 825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe not found: ID does not exist" containerID="825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"
Apr 16 16:35:49.573503 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573438    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"} err="failed to get container status \"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe\": rpc error: code = NotFound desc = could not find container \"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe\": container with ID starting with 825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe not found: ID does not exist"
Apr 16 16:35:49.573503 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573479    2578 scope.go:117] "RemoveContainer" containerID="bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"
Apr 16 16:35:49.573694 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:35:49.573679    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274\": container with ID starting with bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274 not found: ID does not exist" containerID="bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"
Apr 16 16:35:49.573750 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573697    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"} err="failed to get container status \"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274\": rpc error: code = NotFound desc = could not find container \"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274\": container with ID starting with bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274 not found: ID does not exist"
Apr 16 16:35:49.573750 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573710    2578 scope.go:117] "RemoveContainer" containerID="d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"
Apr 16 16:35:49.573924 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573901    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845"} err="failed to get container status \"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845\": rpc error: code = NotFound desc = could not find container \"d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845\": container with ID starting with d0ad337ac452bd90320565f1b820ae0fe2242bc8b88a6917ec84948062f03845 not found: ID does not exist"
Apr 16 16:35:49.573984 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.573925    2578 scope.go:117] "RemoveContainer" containerID="825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"
Apr 16 16:35:49.574121 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.574103    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe"} err="failed to get container status \"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe\": rpc error: code = NotFound desc = could not find container \"825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe\": container with ID starting with 825e9e24490a418c0845157d2f214447bd0b8a5bbd993676a9b6cc5ba37a52fe not found: ID does not exist"
Apr 16 16:35:49.574179 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.574122    2578 scope.go:117] "RemoveContainer" containerID="bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"
Apr 16 16:35:49.574345 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.574327    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274"} err="failed to get container status \"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274\": rpc error: code = NotFound desc = could not find container \"bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274\": container with ID starting with bf6198a92ce7eae7996f2a1f297d1723552fbd153267ea222210778b9a5d2274 not found: ID does not exist"
Apr 16 16:35:49.613530 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.613495    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d1fee1b-f258-4ef6-989c-8bbb284fce53" (UID: "7d1fee1b-f258-4ef6-989c-8bbb284fce53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:35:49.656076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.656048    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d1fee1b-f258-4ef6-989c-8bbb284fce53-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:35:49.656076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.656076    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvc2v\" (UniqueName: \"kubernetes.io/projected/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kube-api-access-xvc2v\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:35:49.656230 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.656085    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:35:49.656230 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.656094    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d1fee1b-f258-4ef6-989c-8bbb284fce53-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:35:49.801942 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.801912    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"]
Apr 16 16:35:49.807833 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:49.807809    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-996756845-nvhq9"]
Apr 16 16:35:51.108809 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:35:51.108776    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" path="/var/lib/kubelet/pods/7d1fee1b-f258-4ef6-989c-8bbb284fce53/volumes"
Apr 16 16:36:00.821561 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:00.821525    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"]
Apr 16 16:36:00.822071 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:00.821853    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="main" containerID="cri-o://7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3" gracePeriod=30
Apr 16 16:36:00.822071 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:00.821914    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="tokenizer" containerID="cri-o://831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6" gracePeriod=30
Apr 16 16:36:00.830259 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:00.830235    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 16 16:36:00.830543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:00.830520    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main" containerID="cri-o://d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4" gracePeriod=30
Apr 16 16:36:01.539891 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.539851    2578 generic.go:358] "Generic (PLEG): container finished" podID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerID="7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3" exitCode=0
Apr 16 16:36:01.540188 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.540167    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerDied","Data":"7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3"}
Apr 16 16:36:01.679339 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.679317    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:36:01.764430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764334    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-kserve-provision-location\") pod \"7b5d646f-01a1-42d1-87b2-33f760b01385\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") "
Apr 16 16:36:01.764430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764388    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-model-cache\") pod \"7b5d646f-01a1-42d1-87b2-33f760b01385\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") "
Apr 16 16:36:01.764430 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764420    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-home\") pod \"7b5d646f-01a1-42d1-87b2-33f760b01385\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") "
Apr 16 16:36:01.764750 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764486    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjspz\" (UniqueName: \"kubernetes.io/projected/7b5d646f-01a1-42d1-87b2-33f760b01385-kube-api-access-pjspz\") pod \"7b5d646f-01a1-42d1-87b2-33f760b01385\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") "
Apr 16 16:36:01.764750 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764508    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5d646f-01a1-42d1-87b2-33f760b01385-tls-certs\") pod \"7b5d646f-01a1-42d1-87b2-33f760b01385\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") "
Apr 16 16:36:01.764750 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764585    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-dshm\") pod \"7b5d646f-01a1-42d1-87b2-33f760b01385\" (UID: \"7b5d646f-01a1-42d1-87b2-33f760b01385\") "
Apr 16 16:36:01.764909 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764746    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b5d646f-01a1-42d1-87b2-33f760b01385" (UID: "7b5d646f-01a1-42d1-87b2-33f760b01385"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:01.764909 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.764850    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-home" (OuterVolumeSpecName: "home") pod "7b5d646f-01a1-42d1-87b2-33f760b01385" (UID: "7b5d646f-01a1-42d1-87b2-33f760b01385"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:01.765124 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.765104    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-model-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:01.765189 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.765129    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-home\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:01.766833 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.766808    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-dshm" (OuterVolumeSpecName: "dshm") pod "7b5d646f-01a1-42d1-87b2-33f760b01385" (UID: "7b5d646f-01a1-42d1-87b2-33f760b01385"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:01.767200 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.767168    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5d646f-01a1-42d1-87b2-33f760b01385-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b5d646f-01a1-42d1-87b2-33f760b01385" (UID: "7b5d646f-01a1-42d1-87b2-33f760b01385"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:36:01.767200 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.767188    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5d646f-01a1-42d1-87b2-33f760b01385-kube-api-access-pjspz" (OuterVolumeSpecName: "kube-api-access-pjspz") pod "7b5d646f-01a1-42d1-87b2-33f760b01385" (UID: "7b5d646f-01a1-42d1-87b2-33f760b01385"). InnerVolumeSpecName "kube-api-access-pjspz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:36:01.828617 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.828583    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b5d646f-01a1-42d1-87b2-33f760b01385" (UID: "7b5d646f-01a1-42d1-87b2-33f760b01385"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:01.866258 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.866228    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:01.866258 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.866257    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjspz\" (UniqueName: \"kubernetes.io/projected/7b5d646f-01a1-42d1-87b2-33f760b01385-kube-api-access-pjspz\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:01.866473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.866268    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b5d646f-01a1-42d1-87b2-33f760b01385-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:01.866473 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.866279    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b5d646f-01a1-42d1-87b2-33f760b01385-dshm\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:01.984891 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:01.984871    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:36:02.068178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068117    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kserve-provision-location\") pod \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") "
Apr 16 16:36:02.068178 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068169    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-tmp\") pod \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") "
Apr 16 16:36:02.068332 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068206    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79xlv\" (UniqueName: \"kubernetes.io/projected/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kube-api-access-79xlv\") pod \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") "
Apr 16 16:36:02.068332 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068225    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tls-certs\") pod \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") "
Apr 16 16:36:02.068332 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068274    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-uds\") pod \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") "
Apr 16 16:36:02.068332 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068329    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-cache\") pod \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\" (UID: \"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522\") "
Apr 16 16:36:02.068632 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068598    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" (UID: "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:02.068759 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068621    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" (UID: "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:02.068759 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.068679    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" (UID: "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:02.069032 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.069001    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" (UID: "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Apr 16 16:36:02.070382 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.070356    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" (UID: "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Apr 16 16:36:02.070482 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.070394    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kube-api-access-79xlv" (OuterVolumeSpecName: "kube-api-access-79xlv") pod "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" (UID: "35d7bf3d-6677-4d4b-8ce5-6c0310fe8522"). InnerVolumeSpecName "kube-api-access-79xlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Apr 16 16:36:02.169690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.169661    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-tmp\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:02.169690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.169692    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79xlv\" (UniqueName: \"kubernetes.io/projected/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kube-api-access-79xlv\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:02.169813 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.169704    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tls-certs\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:02.169813 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.169714    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-uds\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:02.169813 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.169721    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-tokenizer-cache\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:02.169813 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.169729    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522-kserve-provision-location\") on node \"ip-10-0-134-41.ec2.internal\" DevicePath \"\""
Apr 16 16:36:02.546008 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.545985    2578 generic.go:358] "Generic (PLEG): container finished" podID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerID="831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6" exitCode=0
Apr 16 16:36:02.546114 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.546052    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"
Apr 16 16:36:02.546114 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.546062    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerDied","Data":"831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6"}
Apr 16 16:36:02.546114 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.546102    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf" event={"ID":"35d7bf3d-6677-4d4b-8ce5-6c0310fe8522","Type":"ContainerDied","Data":"7a7304cdb1debff0f29ea13d399bf150bf145d32185b9108e2fe3244e52de2a3"}
Apr 16 16:36:02.546248 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.546116    2578 scope.go:117] "RemoveContainer" containerID="831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6"
Apr 16 16:36:02.547875 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.547857    2578 generic.go:358] "Generic (PLEG): container finished" podID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerID="d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4" exitCode=0
Apr 16 16:36:02.547958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.547909    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"7b5d646f-01a1-42d1-87b2-33f760b01385","Type":"ContainerDied","Data":"d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4"}
Apr 16 16:36:02.547958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.547930    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"7b5d646f-01a1-42d1-87b2-33f760b01385","Type":"ContainerDied","Data":"cc2ca2c74f0b4aa9f9b8e5217c4be2fbdbcc6098759394add2f9d4fcb6412f62"}
Apr 16 16:36:02.547958 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.547937    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Apr 16 16:36:02.556839 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.556822    2578 scope.go:117] "RemoveContainer" containerID="7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3"
Apr 16 16:36:02.564563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.564546    2578 scope.go:117] "RemoveContainer" containerID="ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21"
Apr 16 16:36:02.570351 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.570326    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"]
Apr 16 16:36:02.573045 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.573019    2578 scope.go:117] "RemoveContainer" containerID="831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6"
Apr 16 16:36:02.573302 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:36:02.573266    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6\": container with ID starting with 831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6 not found: ID does not exist" containerID="831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6"
Apr 16 16:36:02.573366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.573311    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6"} err="failed to get container status \"831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6\": rpc error: code = NotFound desc = could not find container \"831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6\": container with ID starting with 831756daff1bade9577b1536b456e8c4406818929ae78eb88896fccad0ba75c6 not found: ID does not exist"
Apr 16 16:36:02.573366 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.573333    2578 scope.go:117] "RemoveContainer" containerID="7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3"
Apr 16 16:36:02.573621 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:36:02.573600    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3\": container with ID starting with 7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3 not found: ID does not exist" containerID="7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3"
Apr 16 16:36:02.573689 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.573629    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3"} err="failed to get container status \"7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3\": rpc error: code = NotFound desc = could not find container \"7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3\": container with ID starting with 7232d53c6d31e348c7f53064a13e3e7f9e0a3d24bb84407e8c5440d31ac0eca3 not found: ID does not exist"
Apr 16 16:36:02.573689 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.573645    2578 scope.go:117] "RemoveContainer" containerID="ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21"
Apr 16 16:36:02.573834 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:36:02.573813    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21\": container with ID starting with ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21 not found: ID does not exist" containerID="ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21"
Apr 16 16:36:02.573912 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.573842    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21"} err="failed to get container status \"ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21\": rpc error: code = NotFound desc = could not find container \"ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21\": container with ID starting with ee4ca1e0ea681d32c342ed9456b1e4bf53c5322e3befa2784b69c70c02b5af21 not found: ID does not exist"
Apr 16 16:36:02.573912 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.573864    2578 scope.go:117] "RemoveContainer" containerID="d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4"
Apr 16 16:36:02.574102 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.574079    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc4e643bc258191ffc517a31cd1d0ddd27-kserve-router-sche75cnf"]
Apr 16 16:36:02.584053 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.584031    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 16 16:36:02.588122 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.588103    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Apr 16 16:36:02.592284 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.592264    2578 scope.go:117] "RemoveContainer" containerID="55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e"
Apr 16 16:36:02.657223 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.657203    2578 scope.go:117] "RemoveContainer" containerID="d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4"
Apr 16 16:36:02.657538 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:36:02.657519    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4\": container with ID starting with d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4 not found: ID does not exist" containerID="d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4"
Apr 16 16:36:02.657593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.657547    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4"} err="failed to get container status \"d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4\": rpc error: code = NotFound desc = could not find container \"d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4\": container with ID starting with d6916606e667c6a6014360a6fa1bd90661257c895df39ff32d1b09df6b28cbb4 not found: ID does not exist"
Apr 16 16:36:02.657593 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.657564    2578 scope.go:117] "RemoveContainer" containerID="55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e"
Apr 16 16:36:02.657811 ip-10-0-134-41 kubenswrapper[2578]: E0416 16:36:02.657795    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e\": container with ID starting with 55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e not found: ID does not exist" containerID="55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e"
Apr 16 16:36:02.657864 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:02.657819    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e"} err="failed to get container status \"55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e\": rpc error: code = NotFound desc = could not find container \"55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e\": container with ID starting with 55f2daa84ed3d36a597601f0b0d5be0ddf4aa29a7ae980bb348b63c6f6a6769e not found: ID does not exist"
Apr 16 16:36:03.108089 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:03.108061    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" path="/var/lib/kubelet/pods/35d7bf3d-6677-4d4b-8ce5-6c0310fe8522/volumes"
Apr 16 16:36:03.108509 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:03.108494    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" path="/var/lib/kubelet/pods/7b5d646f-01a1-42d1-87b2-33f760b01385/volumes"
Apr 16 16:36:19.228946 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:19.228919    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:36:19.235547 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:19.235527    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:36:36.914389 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:36.914360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-8lbdh_8a19d4af-2b9e-4a5f-ae96-0567d9434a5f/discovery/0.log"
Apr 16 16:36:37.730227 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:37.730198    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-8lbdh_8a19d4af-2b9e-4a5f-ae96-0567d9434a5f/discovery/0.log"
Apr 16 16:36:38.549220 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:38.549190    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-vmcdg_d52e1c03-0b94-42f8-8233-43563211f01c/manager/0.log"
Apr 16 16:36:38.598190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:38.598170    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-65q4c_f4f3f8cd-567f-495b-ad16-031017f150f4/manager/0.log"
Apr 16 16:36:38.623306 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:38.623286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-88mwq_b9825944-320a-4eeb-90c2-72aa738b68d7/manager/0.log"
Apr 16 16:36:41.531622 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.531590    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xr7j8/must-gather-ldkcs"]
Apr 16 16:36:41.531986 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.531954    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main"
Apr 16 16:36:41.531986 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.531965    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main"
Apr 16 16:36:41.531986 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.531976    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main"
Apr 16 16:36:41.531986 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.531982    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.531990    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="llm-d-routing-sidecar"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.531997    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="llm-d-routing-sidecar"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532004    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="storage-initializer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532009    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="storage-initializer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532015    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="tokenizer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532020    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="tokenizer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532026    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="storage-initializer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532031    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="storage-initializer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532044    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="storage-initializer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532049    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="storage-initializer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532058    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="main"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532062    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="main"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532108    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="main"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532117    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="tokenizer"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532124    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7d1fee1b-f258-4ef6-989c-8bbb284fce53" containerName="llm-d-routing-sidecar"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532131    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="35d7bf3d-6677-4d4b-8ce5-6c0310fe8522" containerName="main"
Apr 16 16:36:41.532139 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.532138    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b5d646f-01a1-42d1-87b2-33f760b01385" containerName="main"
Apr 16 16:36:41.535109 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.535086    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.538282 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.538265    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xr7j8\"/\"default-dockercfg-dtjgr\""
Apr 16 16:36:41.538825 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.538809    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr7j8\"/\"kube-root-ca.crt\""
Apr 16 16:36:41.547076 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.547059    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xr7j8\"/\"openshift-service-ca.crt\""
Apr 16 16:36:41.550764 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.550743    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr7j8/must-gather-ldkcs"]
Apr 16 16:36:41.650890 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.650864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fdfecb9-4f33-465f-b0fa-9610e5bc3698-must-gather-output\") pod \"must-gather-ldkcs\" (UID: \"8fdfecb9-4f33-465f-b0fa-9610e5bc3698\") " pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.650998 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.650957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwtz\" (UniqueName: \"kubernetes.io/projected/8fdfecb9-4f33-465f-b0fa-9610e5bc3698-kube-api-access-qpwtz\") pod \"must-gather-ldkcs\" (UID: \"8fdfecb9-4f33-465f-b0fa-9610e5bc3698\") " pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.752016 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.751989    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fdfecb9-4f33-465f-b0fa-9610e5bc3698-must-gather-output\") pod \"must-gather-ldkcs\" (UID: \"8fdfecb9-4f33-465f-b0fa-9610e5bc3698\") " pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.752118 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.752033    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwtz\" (UniqueName: \"kubernetes.io/projected/8fdfecb9-4f33-465f-b0fa-9610e5bc3698-kube-api-access-qpwtz\") pod \"must-gather-ldkcs\" (UID: \"8fdfecb9-4f33-465f-b0fa-9610e5bc3698\") " pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.752289 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.752272    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fdfecb9-4f33-465f-b0fa-9610e5bc3698-must-gather-output\") pod \"must-gather-ldkcs\" (UID: \"8fdfecb9-4f33-465f-b0fa-9610e5bc3698\") " pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.760424 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.760406    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwtz\" (UniqueName: \"kubernetes.io/projected/8fdfecb9-4f33-465f-b0fa-9610e5bc3698-kube-api-access-qpwtz\") pod \"must-gather-ldkcs\" (UID: \"8fdfecb9-4f33-465f-b0fa-9610e5bc3698\") " pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.844478 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.844395    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr7j8/must-gather-ldkcs"
Apr 16 16:36:41.970262 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.970232    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr7j8/must-gather-ldkcs"]
Apr 16 16:36:41.971974 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:36:41.971943    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fdfecb9_4f33_465f_b0fa_9610e5bc3698.slice/crio-2827ed48dab84febffb4f811f8d25f8f5975df6a1fac05386f1c76c93ee44b8b WatchSource:0}: Error finding container 2827ed48dab84febffb4f811f8d25f8f5975df6a1fac05386f1c76c93ee44b8b: Status 404 returned error can't find the container with id 2827ed48dab84febffb4f811f8d25f8f5975df6a1fac05386f1c76c93ee44b8b
Apr 16 16:36:41.973571 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:41.973555    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Apr 16 16:36:42.708012 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:42.707974    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr7j8/must-gather-ldkcs" event={"ID":"8fdfecb9-4f33-465f-b0fa-9610e5bc3698","Type":"ContainerStarted","Data":"2827ed48dab84febffb4f811f8d25f8f5975df6a1fac05386f1c76c93ee44b8b"}
Apr 16 16:36:43.715129 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:43.715097    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr7j8/must-gather-ldkcs" event={"ID":"8fdfecb9-4f33-465f-b0fa-9610e5bc3698","Type":"ContainerStarted","Data":"5be77a0036cf98b9c02959cd433e62d694bc2c64d08c8a84123ce0dc46e4d713"}
Apr 16 16:36:43.715129 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:43.715131    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr7j8/must-gather-ldkcs" event={"ID":"8fdfecb9-4f33-465f-b0fa-9610e5bc3698","Type":"ContainerStarted","Data":"3db7935af43721a8b4f90e870b1b95fd1b4dfd7f38301385b93cfec76e9b175c"}
Apr 16 16:36:43.732702 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:43.732632    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-xr7j8/must-gather-ldkcs" podStartSLOduration=1.840880464 podStartE2EDuration="2.732614397s" podCreationTimestamp="2026-04-16 16:36:41 +0000 UTC" firstStartedPulling="2026-04-16 16:36:41.973675623 +0000 UTC m=+2123.369760020" lastFinishedPulling="2026-04-16 16:36:42.865409553 +0000 UTC m=+2124.261493953" observedRunningTime="2026-04-16 16:36:43.730395008 +0000 UTC m=+2125.126479425" watchObservedRunningTime="2026-04-16 16:36:43.732614397 +0000 UTC m=+2125.128698843"
Apr 16 16:36:44.424024 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:44.423989    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mjzrr_f822bab8-379f-4f6a-bffd-cbfca5905808/global-pull-secret-syncer/0.log"
Apr 16 16:36:44.462741 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:44.462710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j5rnw_05ae8e22-0b95-4a1f-9939-147624118297/konnectivity-agent/0.log"
Apr 16 16:36:44.560759 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:44.560733    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-41.ec2.internal_ac3c29e3e9ef322bb0cc9eb018b42ffc/haproxy/0.log"
Apr 16 16:36:48.487283 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:48.487252    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-844548ff4c-vmcdg_d52e1c03-0b94-42f8-8233-43563211f01c/manager/0.log"
Apr 16 16:36:48.570870 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:48.570833    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6ddf9554fc-65q4c_f4f3f8cd-567f-495b-ad16-031017f150f4/manager/0.log"
Apr 16 16:36:48.622271 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:48.622240    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-c7fb4c8d5-88mwq_b9825944-320a-4eeb-90c2-72aa738b68d7/manager/0.log"
Apr 16 16:36:49.606331 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.606293    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ae077b4d-8fc0-48f7-b89b-37d32a5db246/alertmanager/0.log"
Apr 16 16:36:49.630995 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.630966    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ae077b4d-8fc0-48f7-b89b-37d32a5db246/config-reloader/0.log"
Apr 16 16:36:49.652775 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.652749    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ae077b4d-8fc0-48f7-b89b-37d32a5db246/kube-rbac-proxy-web/0.log"
Apr 16 16:36:49.673838 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.673804    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ae077b4d-8fc0-48f7-b89b-37d32a5db246/kube-rbac-proxy/0.log"
Apr 16 16:36:49.697647 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.697620    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ae077b4d-8fc0-48f7-b89b-37d32a5db246/kube-rbac-proxy-metric/0.log"
Apr 16 16:36:49.718558 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.718485    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ae077b4d-8fc0-48f7-b89b-37d32a5db246/prom-label-proxy/0.log"
Apr 16 16:36:49.739204 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.739176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ae077b4d-8fc0-48f7-b89b-37d32a5db246/init-config-reloader/0.log"
Apr 16 16:36:49.867553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.867475    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55987d9d7d-8qs6k_95d7b734-0366-4e59-85ac-abbc5e0a18cd/metrics-server/0.log"
Apr 16 16:36:49.928190 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.928153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9v8pw_2b5bff59-222c-4e8a-a42e-bedbd5295241/node-exporter/0.log"
Apr 16 16:36:49.950543 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.950513    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9v8pw_2b5bff59-222c-4e8a-a42e-bedbd5295241/kube-rbac-proxy/0.log"
Apr 16 16:36:49.975550 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:49.975521    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9v8pw_2b5bff59-222c-4e8a-a42e-bedbd5295241/init-textfile/0.log"
Apr 16 16:36:50.172064 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.171982    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-w6c2l_db13baef-3ba8-4571-b68f-b71473a7aa37/kube-rbac-proxy-main/0.log"
Apr 16 16:36:50.199175 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.199142    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-w6c2l_db13baef-3ba8-4571-b68f-b71473a7aa37/kube-rbac-proxy-self/0.log"
Apr 16 16:36:50.223751 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.223710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-5669946b84-w6c2l_db13baef-3ba8-4571-b68f-b71473a7aa37/openshift-state-metrics/0.log"
Apr 16 16:36:50.490475 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.490417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-78f957474d-rz29f_ab0a2370-e936-4e4f-94aa-9dc341b4ef4c/prometheus-operator/0.log"
Apr 16 16:36:50.506772 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.506715    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-78f957474d-rz29f_ab0a2370-e936-4e4f-94aa-9dc341b4ef4c/kube-rbac-proxy/0.log"
Apr 16 16:36:50.543683 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.543648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-9cb97cd87-cjd4p_614a4262-9bed-4287-bdb0-5d151f9397bc/prometheus-operator-admission-webhook/0.log"
Apr 16 16:36:50.583103 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.583070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c89b586b-trw2s_177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02/telemeter-client/0.log"
Apr 16 16:36:50.611844 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.611812    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c89b586b-trw2s_177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02/reload/0.log"
Apr 16 16:36:50.636762 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:50.636730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55c89b586b-trw2s_177aa2ad-9c6e-40a6-8aa4-b2dd06fdbf02/kube-rbac-proxy/0.log"
Apr 16 16:36:52.858976 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.858948    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5db74f6b9d-2qz9p_5461e083-38c9-41c5-9e80-3fb2f3603562/console/0.log"
Apr 16 16:36:52.897364 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.897331    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"]
Apr 16 16:36:52.904563 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.904538    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:52.907163 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.907139    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"]
Apr 16 16:36:52.970670 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.970631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-lib-modules\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:52.970859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.970686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxpts\" (UniqueName: \"kubernetes.io/projected/0ae1524a-d80c-4a54-9d03-9e91845042b5-kube-api-access-fxpts\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:52.970859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.970743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-podres\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:52.970859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.970793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-proc\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:52.970859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:52.970827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-sys\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.071895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.071864    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-lib-modules\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.071895 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.071897    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fxpts\" (UniqueName: \"kubernetes.io/projected/0ae1524a-d80c-4a54-9d03-9e91845042b5-kube-api-access-fxpts\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.072128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.071923    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-podres\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.072128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.071948    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-proc\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.072128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.071966    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-sys\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.072128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.072022    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-sys\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.072128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.072045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-proc\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.072128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.072056    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-podres\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.072128 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.072045    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ae1524a-d80c-4a54-9d03-9e91845042b5-lib-modules\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.079931 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.079909    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxpts\" (UniqueName: \"kubernetes.io/projected/0ae1524a-d80c-4a54-9d03-9e91845042b5-kube-api-access-fxpts\") pod \"perf-node-gather-daemonset-2fm2n\" (UID: \"0ae1524a-d80c-4a54-9d03-9e91845042b5\") " pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.217749 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.217710    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.368553 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.368525    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"]
Apr 16 16:36:53.369874 ip-10-0-134-41 kubenswrapper[2578]: W0416 16:36:53.369839    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0ae1524a_d80c_4a54_9d03_9e91845042b5.slice/crio-39aed116b320d58597aaf1952c0efd2a26fcf88b54abf7151c169782a52e630b WatchSource:0}: Error finding container 39aed116b320d58597aaf1952c0efd2a26fcf88b54abf7151c169782a52e630b: Status 404 returned error can't find the container with id 39aed116b320d58597aaf1952c0efd2a26fcf88b54abf7151c169782a52e630b
Apr 16 16:36:53.780350 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.780250    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n" event={"ID":"0ae1524a-d80c-4a54-9d03-9e91845042b5","Type":"ContainerStarted","Data":"0e7e991a5a605eaea9f4ecd93b06a69ce3caf0f1de7491f2d81fb6280de04e70"}
Apr 16 16:36:53.780350 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.780306    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n" event={"ID":"0ae1524a-d80c-4a54-9d03-9e91845042b5","Type":"ContainerStarted","Data":"39aed116b320d58597aaf1952c0efd2a26fcf88b54abf7151c169782a52e630b"}
Apr 16 16:36:53.780592 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.780512    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:53.799566 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:53.799527    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n" podStartSLOduration=1.799514177 podStartE2EDuration="1.799514177s" podCreationTimestamp="2026-04-16 16:36:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:36:53.79732705 +0000 UTC m=+2135.193411574" watchObservedRunningTime="2026-04-16 16:36:53.799514177 +0000 UTC m=+2135.195598595"
Apr 16 16:36:54.056022 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:54.055941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2rp8s_8f32934c-e56b-47ea-97f7-fb45fd464676/dns/0.log"
Apr 16 16:36:54.091746 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:54.091724    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2rp8s_8f32934c-e56b-47ea-97f7-fb45fd464676/kube-rbac-proxy/0.log"
Apr 16 16:36:54.196841 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:54.196808    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7c442_2d945238-4c81-4dc5-bf64-145e503a3eb3/dns-node-resolver/0.log"
Apr 16 16:36:54.730690 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:54.730660    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nwd2d_26fd2187-a513-457a-9763-91f0fdff4ad6/node-ca/0.log"
Apr 16 16:36:55.543859 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:55.543832    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-55ff986f96-8lbdh_8a19d4af-2b9e-4a5f-ae96-0567d9434a5f/discovery/0.log"
Apr 16 16:36:56.101701 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:56.101672    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vz6nr_a752880f-261b-4efa-a2db-25265bb0a5bb/serve-healthcheck-canary/0.log"
Apr 16 16:36:56.617004 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:56.616981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8wbj_6371e86d-0f49-4ae0-93a2-7f9eed6363ba/kube-rbac-proxy/0.log"
Apr 16 16:36:56.637786 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:56.637764    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8wbj_6371e86d-0f49-4ae0-93a2-7f9eed6363ba/exporter/0.log"
Apr 16 16:36:56.658405 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:56.658385    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l8wbj_6371e86d-0f49-4ae0-93a2-7f9eed6363ba/extractor/0.log"
Apr 16 16:36:59.213742 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:59.213668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-bfc7f696d-kgvsc_9c839d5e-6305-438c-b620-79acbbf1760d/openshift-lws-operator/0.log"
Apr 16 16:36:59.723841 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:59.723812    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7c68cb4fc8-lvg26_6523b077-a1cc-4863-9be5-4ffacd4a2d0e/manager/0.log"
Apr 16 16:36:59.795079 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:59.795055    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xr7j8/perf-node-gather-daemonset-2fm2n"
Apr 16 16:36:59.799554 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:36:59.799531    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-d6n4q_77254d9f-3abe-49b0-ac5c-a0385b611ddf/server/0.log"
Apr 16 16:37:00.013420 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:00.013349    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-5rtdj_a8121fdb-6482-4d89-8b1c-25d3e39b06d8/manager/0.log"
Apr 16 16:37:00.059004 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:00.058955    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-86cc847c5c-5cmmr_83495ec6-95cf-4e61-bd9d-40ef9973c2d4/seaweedfs/0.log"
Apr 16 16:37:04.672815 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:04.672772    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-64d4d94569-nl8rd_7876d91d-8e38-415b-b557-f51d2d0c907c/migrator/0.log"
Apr 16 16:37:04.691585 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:04.691559    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-64d4d94569-nl8rd_7876d91d-8e38-415b-b557-f51d2d0c907c/graceful-termination/0.log"
Apr 16 16:37:06.051287 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.051259    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52kws_5343bde2-108d-43c0-beaf-40395b901cce/kube-multus-additional-cni-plugins/0.log"
Apr 16 16:37:06.075945 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.075914    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52kws_5343bde2-108d-43c0-beaf-40395b901cce/egress-router-binary-copy/0.log"
Apr 16 16:37:06.111221 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.111185    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52kws_5343bde2-108d-43c0-beaf-40395b901cce/cni-plugins/0.log"
Apr 16 16:37:06.172254 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.172230    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52kws_5343bde2-108d-43c0-beaf-40395b901cce/bond-cni-plugin/0.log"
Apr 16 16:37:06.207513 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.207489    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52kws_5343bde2-108d-43c0-beaf-40395b901cce/routeoverride-cni/0.log"
Apr 16 16:37:06.238941 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.238917    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52kws_5343bde2-108d-43c0-beaf-40395b901cce/whereabouts-cni-bincopy/0.log"
Apr 16 16:37:06.278370 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.278342    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52kws_5343bde2-108d-43c0-beaf-40395b901cce/whereabouts-cni/0.log"
Apr 16 16:37:06.843492 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.843466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xkfxp_ba1e3eb8-7e0a-4f07-add3-8e05de8932bf/kube-multus/0.log"
Apr 16 16:37:06.949921 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.949898    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pbf54_53da8c88-a72c-4cf8-abde-f3a449cbdf3f/network-metrics-daemon/0.log"
Apr 16 16:37:06.978116 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:06.978091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pbf54_53da8c88-a72c-4cf8-abde-f3a449cbdf3f/kube-rbac-proxy/0.log"
Apr 16 16:37:08.184706 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.184676    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-controller/0.log"
Apr 16 16:37:08.204607 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.204582    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/0.log"
Apr 16 16:37:08.214571 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.214547    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovn-acl-logging/1.log"
Apr 16 16:37:08.234438 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.234408    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/kube-rbac-proxy-node/0.log"
Apr 16 16:37:08.257935 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.257902    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/kube-rbac-proxy-ovn-metrics/0.log"
Apr 16 16:37:08.282520 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.282497    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/northd/0.log"
Apr 16 16:37:08.303795 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.303780    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/nbdb/0.log"
Apr 16 16:37:08.328034 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.328006    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/sbdb/0.log"
Apr 16 16:37:08.452030 ip-10-0-134-41 kubenswrapper[2578]: I0416 16:37:08.451965    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9n7wn_169cfa67-f1fa-4803-84bc-2219a6384156/ovnkube-controller/0.log"