Jul 10 18:39:52.915517 ip-10-0-134-57 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:39:53.361428 ip-10-0-134-57 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:53.361428 ip-10-0-134-57 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:39:53.361428 ip-10-0-134-57 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:53.361428 ip-10-0-134-57 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:39:53.361428 ip-10-0-134-57 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:53.364630 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.364467    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:39:53.371761 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371730    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:53.371761 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371763    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371771    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371776    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371781    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371786    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371791    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371797    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371803    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371807    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371812    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371816    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371821    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371825    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371831    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371835    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371839    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371843    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371846    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371851    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371854    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:53.371855 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371858    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371863    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371867    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371871    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371875    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371879    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371883    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371888    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371892    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371897    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371901    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371905    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371908    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371912    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371916    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371919    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371923    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371927    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371930    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:53.372338 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371934    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371938    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371942    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371946    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371949    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371953    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371957    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371961    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371965    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371968    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371972    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371975    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371979    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371982    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371986    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371991    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371995    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.371999    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372003    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372006    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:53.372819 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372010    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372014    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372018    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372022    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372025    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372028    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372032    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372035    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372039    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372043    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372047    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372051    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372054    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372058    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372062    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372067    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372070    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372074    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372078    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372082    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:53.373326 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372086    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372090    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372094    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372101    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372105    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372109    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372115    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372121    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372126    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372130    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372135    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372139    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372156    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372161    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372165    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372169    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372173    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372177    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.372181    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373329    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:53.373801 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373341    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373347    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373352    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373356    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373361    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373365    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373370    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373374    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373379    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373383    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373387    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373391    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373395    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373399    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373404    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373410    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373416    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373421    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373426    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373430    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:53.374316 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373435    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373440    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373444    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373448    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373453    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373456    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373460    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373464    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373468    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373472    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373476    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373480    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373483    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373487    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373490    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373494    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373498    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373502    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373505    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373508    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:53.374809 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373512    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373517    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373521    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373527    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373532    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373536    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373541    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373545    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373549    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373553    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373557    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373561    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373565    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373568    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373572    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373576    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373579    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373583    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373586    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373590    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:53.375310 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373594    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373598    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373602    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373606    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373609    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373613    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373616    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373620    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373624    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373629    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373632    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373637    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373640    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373644    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373650    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373654    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373658    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373662    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373666    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373671    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:53.375806 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373675    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373678    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373682    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373686    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373692    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373697    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373701    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373706    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373709    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373714    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373718    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373721    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373725    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373729    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373733    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373737    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373740    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.373744    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374232    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:53.376322 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374243    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374246    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374249    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374251    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374254    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374259    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374262    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374265    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374268    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374270    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374273    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374276    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374279    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374282    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374286    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374290    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374294    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374297    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374300    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374303    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:53.376853 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374306    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374308    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374311    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374314    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374316    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374336    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374340    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374342    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374345    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374348    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374350    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374353    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374357    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374360    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374362    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374365    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374368    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374370    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374373    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374376    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:53.377633 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374379    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374382    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374384    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374387    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374389    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374392    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374394    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374397    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374400    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374402    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374405    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374407    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374410    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374412    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374416    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374420    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374424    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374427    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374430    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374433    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:53.378576 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374435    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374438    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374440    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374443    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374446    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374449    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374452    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374454    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374457    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374460    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374463    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374465    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374469    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374478    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374481    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374484    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374487    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374490    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374493    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:53.379417 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374495    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374498    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374502    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374504    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374507    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374510    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374512    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374515    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374523    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374526    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374529    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374531    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374534    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374537    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374539    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374542    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374544    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374547    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.374550    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374636    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374645    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:39:53.380112 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374652    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374656    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374661    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374664    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374669    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374674    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374677    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374681    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374684    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374688    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374691    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374694    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374697    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374700    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374703    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374706    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374711    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374713    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374716    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374722    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374725    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374729    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374732    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:39:53.380692 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374735    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374739    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374742    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374745    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374748    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374751    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374754    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374758    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374762    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374765    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374768    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374771    2561 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374774    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374780    2561 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374783    2561 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374787    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374790    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374793    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374797    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374800    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374804    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374807    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374810    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374814    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374817    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:39:53.381326 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374822    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374825    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374827    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374832    2561 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374836    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374839    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374842    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374845    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374848    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374852    2561 flags.go:64] FLAG: --help="false"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374855    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374858    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374861    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374864    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374868    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374871    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374874    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374877    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374880    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374883    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374886    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374889    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374892    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374895    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:39:53.381959 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374900    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374903    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374906    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374909    2561 flags.go:64] FLAG: --lock-file=""
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374912    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374915    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374919    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374924    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374927    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374930    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374933    2561 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374937    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374940    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374943    2561 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374946    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374950    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374953    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374957    2561 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374961    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374964    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374967    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374971    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374974    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374977    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374980    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:39:53.382559 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374989    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374992    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374996    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.374999    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375002    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375008    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375011    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375014    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375020    2561 flags.go:64] FLAG: --port="10250"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375023    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375026    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e8f93ffe1a103b57"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375029    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375032    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375035    2561 flags.go:64] FLAG: --register-node="true"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375038    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375042    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375045    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375048    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375052    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375056    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375059    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375063    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375066    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375068    2561 flags.go:64] FLAG: --runonce="false"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375071    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:39:53.383182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375074    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375078    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375081    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375084    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375087    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375090    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375093    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375096    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375099    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375102    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375105    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375108    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375111    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375114    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375119    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375123    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375127    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375132    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375134    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375138    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375156    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375162    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375165    2561 flags.go:64] FLAG: --v="2"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375170    2561 flags.go:64] FLAG: --version="false"
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375174    2561 flags.go:64] FLAG: --vmodule=""
Jul 10 18:39:53.383867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375180    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.375184    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375333    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375338    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375341    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375345    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375348    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375351    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375354    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375357    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375360    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375362    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375365    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375368    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375370    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375373    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375376    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375378    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375381    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375383    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:53.384542 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375386    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375389    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375394    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375397    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375401    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375405    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375408    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375411    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375413    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375416    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375419    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375421    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375425    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375427    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375431    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375433    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375436    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375438    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375441    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:53.385047 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375445    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375448    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375452    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375455    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375457    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375461    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375463    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375466    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375469    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375471    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375474    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375477    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375479    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375482    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375484    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375488    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375491    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375494    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375498    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:53.385607 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375500    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375503    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375505    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375508    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375511    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375513    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375516    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375519    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375522    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375526    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375530    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375533    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375536    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375539    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375542    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375544    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375547    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375549    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375552    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:53.386070 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375554    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375557    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375560    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375563    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375565    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375568    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375570    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375573    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375575    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375579    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375582    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375585    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375587    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375590    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375593    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375596    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375598    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375601    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375603    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375606    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:53.386568 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375609    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375611    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375614    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.375616    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.376272    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.383575    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.383595    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.383616    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.383622    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383683    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383689    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383692    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383695    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383698    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383701    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:53.387048 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383704    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383707    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383710    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383713    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383715    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383718    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383721    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383724    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383727    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383730    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383733    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383736    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383738    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383741    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383744    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383748    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383751    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383755    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383758    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383761    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:53.387486 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383763    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383766    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383769    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383772    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383775    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383778    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383780    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383783    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383785    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383788    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383791    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383793    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383796    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383799    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383801    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383804    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383807    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383810    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383812    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:53.388235 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383815    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383818    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383821    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383823    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383826    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383829    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383832    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383835    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383838    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383840    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383843    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383846    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383848    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383852    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383855    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383858    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383861    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383864    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383868    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:53.388738 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383871    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383874    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383877    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383879    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383882    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383885    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383888    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383890    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383893    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383895    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383898    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383901    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383904    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383907    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383910    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383912    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383915    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383918    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383921    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383923    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383926    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:53.389240 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383928    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383931    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383933    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383936    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383939    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383941    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383944    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383946    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383950    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383954    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383957    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383960    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383963    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.383966    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:53.389821 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.383971    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384092    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384098    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384101    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384103    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384107    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384109    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384112    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384115    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384119    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384122    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384125    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384127    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384130    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384133    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384135    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384138    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384160    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384164    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384167    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:53.390223 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384169    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384172    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384175    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384178    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384180    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384183    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384186    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384188    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384191    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384194    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384197    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384199    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384202    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384206    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384208    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384211    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384213    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384216    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384219    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:53.390804 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384222    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384224    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384227    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384230    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384233    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384236    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384238    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384241    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384244    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384246    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384249    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384253    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384256    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384259    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384262    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384265    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384268    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384270    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384273    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384275    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:53.391308 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384278    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384281    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384283    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384286    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384289    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384292    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384295    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384299    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384302    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384305    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384308    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384310    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384313    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384315    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384318    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384321    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384323    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384326    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384328    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384331    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:53.391784 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384333    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384336    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384340    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384343    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384346    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384349    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384351    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384355    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384357    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384360    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384362    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384365    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384368    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384371    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384373    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384376    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384379    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384382    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384388    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:53.392279 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384390    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:53.392794 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:53.384393    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:53.392794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.384398    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:53.392794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.384546    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:39:53.392794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.386934    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:39:53.392794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.392611    2561 server.go:1019] "Starting client certificate rotation"
Jul 10 18:39:53.392794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.392732    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:39:53.392794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.392788    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:39:53.395462 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.395444    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:39:53.396661 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.396632    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:39:53.407289 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.407268    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:39:53.410732 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.410710    2561 log.go:25] "Validated CRI v1 image API"
Jul 10 18:39:53.412474 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.412459    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:39:53.415735 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.415708    2561 fs.go:135] Filesystem UUIDs: map[5abe4567-ec49-4cc3-8601-4711f75c76d8:/dev/nvme0n1p3 6c924d88-fb58-4f77-a4fe-325232cfe64e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 10 18:39:53.415796 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.415731    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:39:53.422029 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.421900    2561 manager.go:216] Machine: {Timestamp:2026-07-10 18:39:53.420566067 +0000 UTC m=+0.374854125 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114986 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b63721101773ab38468a1c2180799 SystemUUID:ec2b6372-1101-773a-b384-68a1c2180799 BootID:b7461e8d-4d8b-49d5-b122-754adaa2fb9b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d3:24:80:46:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d3:24:80:46:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:ad:a5:42:a1:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:39:53.422029 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.422020    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:39:53.422205 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.422192    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:39:53.423264 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.423239    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:39:53.425457 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.425424    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:39:53.425619 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.425465    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:39:53.426791 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.426776    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:39:53.426827 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.426795    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:39:53.426827 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.426803    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:39:53.427705 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.427690    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:39:53.428861 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.428846    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:39:53.432097 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432077    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:39:53.432271 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432261    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:39:53.432316 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432296    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:39:53.432316 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432313    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:39:53.432367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432331    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:39:53.432367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432348    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:39:53.432367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432360    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:39:53.432915 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432900    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:39:53.432973 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.432921    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:39:53.437108 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.437076    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:39:53.438872 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.438856    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:39:53.438925 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.438889    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:39:53.439712 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439697    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:39:53.439797 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439720    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:39:53.439797 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439730    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:39:53.439797 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439739    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:39:53.439797 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439751    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:39:53.439797 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439760    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:39:53.439930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439813    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:39:53.439930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439822    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:39:53.439930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439833    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:39:53.439930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439842    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:39:53.439930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439854    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:39:53.439930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.439865    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:39:53.440710 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.440696    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:39:53.440710 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.440710    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:39:53.444186 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.444139    2561 server.go:1268] "Started kubelet"
Jul 10 18:39:53.444329 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.444232    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:39:53.444454 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.444412    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:39:53.444504 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.444477    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:39:53.445034 ip-10-0-134-57 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:39:53.446217 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.446192    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:39:53.446332 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.446315    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 18:39:53.446389 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.446368    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 18:39:53.446475 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.446453    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 18:39:53.446530 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.446478    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:39:53.450620 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.450597    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:39:53.451637 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.451625    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:39:53.451695 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.451649    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:39:53.451753 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.451713    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:53.452502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452321    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:39:53.452502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452339    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:39:53.452502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452354    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:39:53.452502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452493    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:39:53.452502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452503    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:39:53.452827 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452638    2561 factory.go:55] Registering systemd factory
Jul 10 18:39:53.452827 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452655    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:39:53.452896 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452873    2561 factory.go:153] Registering CRI-O factory
Jul 10 18:39:53.452896 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452889    2561 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:39:53.452951 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452934    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:39:53.453003 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452953    2561 factory.go:103] Registering Raw factory
Jul 10 18:39:53.453003 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.452964    2561 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:39:53.453587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.453560    2561 manager.go:349] Starting recovery of all containers
Jul 10 18:39:53.453860 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.452265    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-57.ec2.internal.18c101b277fa01b6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-57.ec2.internal,UID:ip-10-0-134-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-57.ec2.internal,},FirstTimestamp:2026-07-10 18:39:53.444106678 +0000 UTC m=+0.398394815,LastTimestamp:2026-07-10 18:39:53.444106678 +0000 UTC m=+0.398394815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-57.ec2.internal,}"
Jul 10 18:39:53.459876 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.459838    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 10 18:39:53.460053 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.459850    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 10 18:39:53.463402 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.463374    2561 manager.go:354] Recovery completed
Jul 10 18:39:53.468843 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.468822    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:53.474702 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.474679    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5lf46"
Jul 10 18:39:53.476799 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.476777    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:53.476930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.476811    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:53.476930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.476825    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:53.477409 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.477394    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:39:53.477409 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.477406    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:39:53.477529 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.477411    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:53.477529 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.477424    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:39:53.478633 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.478612    2561 policy_none.go:49] "None policy: Start"
Jul 10 18:39:53.478633 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.478634    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:39:53.478780 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.478647    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:39:53.478949 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.478933    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:53.478990 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.478962    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:53.478990 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.478972    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:53.480008 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.479996    2561 policy_none.go:47] "Start"
Jul 10 18:39:53.480400 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.480324    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-57.ec2.internal.18c101b279ecce32  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-57.ec2.internal,UID:ip-10-0-134-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-57.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-57.ec2.internal,},FirstTimestamp:2026-07-10 18:39:53.476795954 +0000 UTC m=+0.431084012,LastTimestamp:2026-07-10 18:39:53.476795954 +0000 UTC m=+0.431084012,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-57.ec2.internal,}"
Jul 10 18:39:53.484055 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.484034    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5lf46"
Jul 10 18:39:53.489572 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.489354    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-57.ec2.internal.18c101b279ed2728  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-57.ec2.internal,UID:ip-10-0-134-57.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-57.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-57.ec2.internal,},FirstTimestamp:2026-07-10 18:39:53.476818728 +0000 UTC m=+0.431106785,LastTimestamp:2026-07-10 18:39:53.476818728 +0000 UTC m=+0.431106785,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-57.ec2.internal,}"
Jul 10 18:39:53.516755 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.516733    2561 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.516784    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.516797    2561 server.go:87] "Starting device plugin registration server"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.517255    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.517271    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.517378    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.517471    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.517480    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.518088    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.518157    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.533633    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.534916    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.534948    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.534968    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.535002    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 18:39:53.541673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.537749    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:53.617907 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.617823    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:53.618788 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.618770    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:53.618884 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.618802    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:53.618884 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.618815    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:53.618884 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.618838    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.628890 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.628862    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.628890 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.628893    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-57.ec2.internal\": node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:53.635708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.635673    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal"]
Jul 10 18:39:53.635828 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.635744    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:53.637307 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.637289    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:53.637375 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.637323    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:53.637375 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.637336    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:53.639543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.639529    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:53.639659 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.639644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.639697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.639680    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:53.640331 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.640311    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:53.640331 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.640325    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:53.640331 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.640339    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:53.640533 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.640350    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:53.640533 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.640351    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:53.640533 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.640376    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:53.642490 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.642472    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.642584 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.642501    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:53.643316 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.643298    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:53.643408 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.643333    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:53.643408 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.643348    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:53.652890 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.652860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40921e7d9f497f62d3e2dfd9e8ed1bc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal\" (UID: \"40921e7d9f497f62d3e2dfd9e8ed1bc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.652890 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.652892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40921e7d9f497f62d3e2dfd9e8ed1bc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal\" (UID: \"40921e7d9f497f62d3e2dfd9e8ed1bc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.653081 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.652914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc0950c7cf4a4407bce2f7a337f490f9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.ec2.internal\" (UID: \"fc0950c7cf4a4407bce2f7a337f490f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.653081 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.652972    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:53.656866 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.656849    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-57.ec2.internal\" not found" node="ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.661738 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.661717    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-57.ec2.internal\" not found" node="ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.753364 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.753333    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:53.753364 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.753360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40921e7d9f497f62d3e2dfd9e8ed1bc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal\" (UID: \"40921e7d9f497f62d3e2dfd9e8ed1bc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.753587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.753387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc0950c7cf4a4407bce2f7a337f490f9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.ec2.internal\" (UID: \"fc0950c7cf4a4407bce2f7a337f490f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.753587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.753406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40921e7d9f497f62d3e2dfd9e8ed1bc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal\" (UID: \"40921e7d9f497f62d3e2dfd9e8ed1bc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.753587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.753442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40921e7d9f497f62d3e2dfd9e8ed1bc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal\" (UID: \"40921e7d9f497f62d3e2dfd9e8ed1bc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.753587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.753450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40921e7d9f497f62d3e2dfd9e8ed1bc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal\" (UID: \"40921e7d9f497f62d3e2dfd9e8ed1bc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.753587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.753480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc0950c7cf4a4407bce2f7a337f490f9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-57.ec2.internal\" (UID: \"fc0950c7cf4a4407bce2f7a337f490f9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.854536 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.854492    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:53.954823 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:53.954742    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:53.960328 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.960293    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:53.965878 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:53.965838    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:54.055546 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:54.055506    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:54.159247 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:54.159205    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:54.260341 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:54.260247    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:54.361125 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:54.361085    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:54.391368 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.391335    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:39:54.391852 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.391520    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:39:54.461242 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:54.461196    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:54.489196 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.489138    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:34:53 +0000 UTC" deadline="2028-01-26 11:53:21.556204237 +0000 UTC"
Jul 10 18:39:54.489196 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.489190    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13553h13m27.067017964s"
Jul 10 18:39:54.562360 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:54.562280    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:54.570526 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.570491    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:54.571968 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.571938    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:54.572105 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.571979    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:54.572105 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.571992    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:54.610654 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.610626    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:54.638665 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:54.638630    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40921e7d9f497f62d3e2dfd9e8ed1bc8.slice/crio-aa75545cbb5c69e91264451dadc8881453c5ab99e119b202fa8fc90c6aa393d1 WatchSource:0}: Error finding container aa75545cbb5c69e91264451dadc8881453c5ab99e119b202fa8fc90c6aa393d1: Status 404 returned error can't find the container with id aa75545cbb5c69e91264451dadc8881453c5ab99e119b202fa8fc90c6aa393d1
Jul 10 18:39:54.648129 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.648108    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:39:54.650424 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:54.650398    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0950c7cf4a4407bce2f7a337f490f9.slice/crio-248a14846a0250a6fc2c606021f9e26fd4147de6e89829412b9c023074195b3f WatchSource:0}: Error finding container 248a14846a0250a6fc2c606021f9e26fd4147de6e89829412b9c023074195b3f: Status 404 returned error can't find the container with id 248a14846a0250a6fc2c606021f9e26fd4147de6e89829412b9c023074195b3f
Jul 10 18:39:54.663257 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:54.663213    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-57.ec2.internal\" not found"
Jul 10 18:39:54.710724 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.710688    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:54.754367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.754329    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:54.772291 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.772262    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:39:54.774746 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.774727    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal"
Jul 10 18:39:54.784676 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:54.784652    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:39:55.026272 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.026185    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:55.433980 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.433895    2561 apiserver.go:52] "Watching apiserver"
Jul 10 18:39:55.443282 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.443235    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:55.443674 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.443653    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:39:55.444160 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.444126    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-4sbhh","openshift-network-diagnostics/network-check-target-fjdnw","openshift-ovn-kubernetes/ovnkube-node-nltht","kube-system/konnectivity-agent-zw7sz","kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2","openshift-dns/node-resolver-hhb4q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal","openshift-multus/multus-kkdrl","openshift-network-operator/iptables-alerter-f267f","kube-system/global-pull-secret-syncer-krjlw","openshift-cluster-node-tuning-operator/tuned-z58qt","openshift-image-registry/node-ca-8cvcp","openshift-multus/multus-additional-cni-plugins-2gkfp"]
Jul 10 18:39:55.447349 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.446995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.449562 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.449530    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.449684 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.449642    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:39:55.450576 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.450219    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:39:55.450576 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.450342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4pcpt\""
Jul 10 18:39:55.450576 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.450414    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:39:55.453289 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.452054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.453289 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.452984    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:39:55.455023 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.454997    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:39:55.455129 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.455119    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.455268 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.455248    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:39:55.455899 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.455353    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:39:55.455899 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.455468    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:39:55.455899 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.455581    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4c4jp\""
Jul 10 18:39:55.455899 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.455679    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:39:55.455899 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.455791    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:39:55.457853 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.457446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.458001 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.457977    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9vdrs\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.459041    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.459319    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.460716    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.460997    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4jf28\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.461328    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.461366    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.461551    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.461551    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:39:55.462397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.461883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.463833 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.463001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.465065 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.465045    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.467694 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.467657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:55.467844 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.467729    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:39:55.468347 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.468324    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:39:55.468446 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.468384    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:39:55.469210 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.468873    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sjcsf\""
Jul 10 18:39:55.469210 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-run-netns\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.469371 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-var-lib-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.469371 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469349    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:39:55.469371 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-cni-netd\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.469543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-kubelet-config\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.469543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-run-ovn-kubernetes\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.469543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469458    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee32311a-4aa2-4168-bffd-f355e8dae244-hosts-file\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.469543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-cni-bin\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.469543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee32311a-4aa2-4168-bffd-f355e8dae244-tmp-dir\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.469804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469575    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.469804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:39:55.469804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r6t5t\""
Jul 10 18:39:55.469804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovn-node-metrics-cert\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-log-socket\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-kubelet\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovnkube-config\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-systemd-units\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469952    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.469995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-slash\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470049 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-ovn\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frn4\" (UniqueName: \"kubernetes.io/projected/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-kube-api-access-9frn4\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dfd4ef1f-f28c-4979-bc40-20da7f09a697-agent-certs\") pod \"konnectivity-agent-zw7sz\" (UID: \"dfd4ef1f-f28c-4979-bc40-20da7f09a697\") " pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-dbus\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470176    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-systemd\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470200    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tq8lp\""
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-node-log\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-env-overrides\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470362    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovnkube-script-lib\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4ef1f-f28c-4979-bc40-20da7f09a697-konnectivity-ca\") pod \"konnectivity-agent-zw7sz\" (UID: \"dfd4ef1f-f28c-4979-bc40-20da7f09a697\") " pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470246    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470427    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-etc-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.470512 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.470467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcgvr\" (UniqueName: \"kubernetes.io/projected/ee32311a-4aa2-4168-bffd-f355e8dae244-kube-api-access-xcgvr\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.472805 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.472673    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:55.472805 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.472751    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:39:55.479123 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.475938    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.479123 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.477947    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:39:55.479123 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.478359    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:39:55.479123 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.478600    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:39:55.479123 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.478611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wlccj\""
Jul 10 18:39:55.479888 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.479624    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:39:55.479888 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.479743    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b8889\""
Jul 10 18:39:55.480683 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.480597    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:39:55.533929 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.533896    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tq8m"
Jul 10 18:39:55.540582 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.540521    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal" event={"ID":"fc0950c7cf4a4407bce2f7a337f490f9","Type":"ContainerStarted","Data":"248a14846a0250a6fc2c606021f9e26fd4147de6e89829412b9c023074195b3f"}
Jul 10 18:39:55.543107 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.543070    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal" event={"ID":"40921e7d9f497f62d3e2dfd9e8ed1bc8","Type":"ContainerStarted","Data":"aa75545cbb5c69e91264451dadc8881453c5ab99e119b202fa8fc90c6aa393d1"}
Jul 10 18:39:55.544871 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.544840    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tq8m"
Jul 10 18:39:55.553744 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.553708    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:39:55.571493 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-conf-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.571670 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-run\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.571670 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-socket-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.571670 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-host\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.571670 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-serviceca\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.571670 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysconfig\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-systemd\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-lib-modules\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571775    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-tuned\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-os-release\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-os-release\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-hostroot\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-modprobe-d\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-cni-bin\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-kubelet\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-etc-kubernetes\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.571953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tn2q\" (UniqueName: \"kubernetes.io/projected/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-kube-api-access-8tn2q\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovn-node-metrics-cert\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-cni-bin\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.571996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fb85dc-3804-4c03-9735-fe08cd156f0d-tmp\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d968d33f-746c-4f85-8118-64bf2e41a3ee-iptables-alerter-script\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572122    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-kubelet\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-kubelet\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-k8s-cni-cncf-io\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysctl-conf\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-cnibin\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572298    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm29s\" (UniqueName: \"kubernetes.io/projected/e48d716c-b5ee-4a31-b532-b815b3b58626-kube-api-access-dm29s\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-registration-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-slash\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-ovn\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572455    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-slash\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.572572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-cni-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-ovn\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-multus-certs\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-kubernetes\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-node-log\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovnkube-script-lib\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4ef1f-f28c-4979-bc40-20da7f09a697-konnectivity-ca\") pod \"konnectivity-agent-zw7sz\" (UID: \"dfd4ef1f-f28c-4979-bc40-20da7f09a697\") " pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-node-log\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-netns\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcgvr\" (UniqueName: \"kubernetes.io/projected/ee32311a-4aa2-4168-bffd-f355e8dae244-kube-api-access-xcgvr\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572898    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-var-lib-kubelet\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-device-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.573450 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.572989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-run-netns\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-var-lib-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-run-netns\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-cni-netd\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-kubelet-config\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-var-lib-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-cni-bin\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-kubelet-config\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-host\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-cni-netd\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqkz\" (UniqueName: \"kubernetes.io/projected/13fb85dc-3804-4c03-9735-fe08cd156f0d-kube-api-access-fhqkz\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-run-ovn-kubernetes\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovnkube-script-lib\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-run-ovn-kubernetes\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee32311a-4aa2-4168-bffd-f355e8dae244-hosts-file\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-socket-dir-parent\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.574311 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573519    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee32311a-4aa2-4168-bffd-f355e8dae244-hosts-file\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-sys-fs\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpj2\" (UniqueName: \"kubernetes.io/projected/d968d33f-746c-4f85-8118-64bf2e41a3ee-kube-api-access-4lpj2\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee32311a-4aa2-4168-bffd-f355e8dae244-tmp-dir\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-cnibin\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-cni-multus\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrc5\" (UniqueName: \"kubernetes.io/projected/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-kube-api-access-scrc5\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-system-cni-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-cni-binary-copy\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-daemon-config\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573883    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-log-socket\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573907    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovnkube-config\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee32311a-4aa2-4168-bffd-f355e8dae244-tmp-dir\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.573936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.575180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqw2\" (UniqueName: \"kubernetes.io/projected/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-kube-api-access-grqw2\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcnzj\" (UniqueName: \"kubernetes.io/projected/e8345347-19c1-46c8-8524-b906e0a9fbbf-kube-api-access-kcnzj\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574069    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-systemd-units\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.574088    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574109    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-systemd-units\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-log-socket\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frn4\" (UniqueName: \"kubernetes.io/projected/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-kube-api-access-9frn4\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574260    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dfd4ef1f-f28c-4979-bc40-20da7f09a697-agent-certs\") pod \"konnectivity-agent-zw7sz\" (UID: \"dfd4ef1f-f28c-4979-bc40-20da7f09a697\") " pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.574304    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret podName:3c91b6f5-d69d-47d5-9782-18d0a40d6f4e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:56.07427689 +0000 UTC m=+3.028564966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret") pod "global-pull-secret-syncer-krjlw" (UID: "3c91b6f5-d69d-47d5-9782-18d0a40d6f4e") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-dbus\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-sys\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-systemd\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-env-overrides\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysctl-d\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.576063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d968d33f-746c-4f85-8118-64bf2e41a3ee-host-slash\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-etc-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574707    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-etc-openvswitch\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.574874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovnkube-config\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.575140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-dbus\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.575244    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dfd4ef1f-f28c-4979-bc40-20da7f09a697-konnectivity-ca\") pod \"konnectivity-agent-zw7sz\" (UID: \"dfd4ef1f-f28c-4979-bc40-20da7f09a697\") " pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.575323    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-run-systemd\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.576869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.576005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-env-overrides\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.577884 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.577856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-ovn-node-metrics-cert\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.582053 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.581995    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dfd4ef1f-f28c-4979-bc40-20da7f09a697-agent-certs\") pod \"konnectivity-agent-zw7sz\" (UID: \"dfd4ef1f-f28c-4979-bc40-20da7f09a697\") " pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.583116 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.583049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcgvr\" (UniqueName: \"kubernetes.io/projected/ee32311a-4aa2-4168-bffd-f355e8dae244-kube-api-access-xcgvr\") pod \"node-resolver-hhb4q\" (UID: \"ee32311a-4aa2-4168-bffd-f355e8dae244\") " pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.585585 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.585561    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frn4\" (UniqueName: \"kubernetes.io/projected/cc3dac8e-46b2-4462-9b80-f1ddd3732eed-kube-api-access-9frn4\") pod \"ovnkube-node-nltht\" (UID: \"cc3dac8e-46b2-4462-9b80-f1ddd3732eed\") " pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-var-lib-kubelet\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-device-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-cni-bin\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-var-lib-kubelet\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-host\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqkz\" (UniqueName: \"kubernetes.io/projected/13fb85dc-3804-4c03-9735-fe08cd156f0d-kube-api-access-fhqkz\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-socket-dir-parent\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-sys-fs\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-etc-selinux\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpj2\" (UniqueName: \"kubernetes.io/projected/d968d33f-746c-4f85-8118-64bf2e41a3ee-kube-api-access-4lpj2\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-cnibin\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-device-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-cni-multus\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.675662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675630    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-sys-fs\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrc5\" (UniqueName: \"kubernetes.io/projected/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-kube-api-access-scrc5\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-system-cni-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-cni-binary-copy\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-daemon-config\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675866    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqw2\" (UniqueName: \"kubernetes.io/projected/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-kube-api-access-grqw2\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcnzj\" (UniqueName: \"kubernetes.io/projected/e8345347-19c1-46c8-8524-b906e0a9fbbf-kube-api-access-kcnzj\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-sys\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.675982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysctl-d\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d968d33f-746c-4f85-8118-64bf2e41a3ee-host-slash\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-conf-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-run\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-socket-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-host\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.676686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-serviceca\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysconfig\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-systemd\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676282    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-lib-modules\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-tuned\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-os-release\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-os-release\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-hostroot\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-conf-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-modprobe-d\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-system-cni-dir\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-kubelet\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-etc-kubernetes\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-kubelet\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676530    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tn2q\" (UniqueName: \"kubernetes.io/projected/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-kube-api-access-8tn2q\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676544    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-daemon-config\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-run\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.677572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676562    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-host\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-cni-multus\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676678    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-cnibin\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.676773    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-socket-dir-parent\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-var-lib-cni-bin\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.676886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-etc-kubernetes\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.676957    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:56.176936666 +0000 UTC m=+3.131224713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677052    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-serviceca\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677113    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-host\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-socket-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677235    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-system-cni-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677331    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-lib-modules\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677421    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-os-release\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-hostroot\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-modprobe-d\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677163    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysconfig\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.678519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-systemd\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-cni-binary-copy\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fb85dc-3804-4c03-9735-fe08cd156f0d-tmp\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d968d33f-746c-4f85-8118-64bf2e41a3ee-iptables-alerter-script\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysctl-d\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-k8s-cni-cncf-io\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.677909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d968d33f-746c-4f85-8118-64bf2e41a3ee-host-slash\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-sys\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-k8s-cni-cncf-io\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysctl-conf\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-cnibin\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm29s\" (UniqueName: \"kubernetes.io/projected/e48d716c-b5ee-4a31-b532-b815b3b58626-kube-api-access-dm29s\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-registration-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d968d33f-746c-4f85-8118-64bf2e41a3ee-iptables-alerter-script\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678611    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-sysctl-conf\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-cnibin\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.679612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-cni-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-multus-certs\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-registration-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-kubernetes\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678818    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-netns\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-kubernetes\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-multus-certs\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.678957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.679059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e8345347-19c1-46c8-8524-b906e0a9fbbf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.679116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-multus-cni-dir\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.679173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-host-run-netns\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.679311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.679340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e48d716c-b5ee-4a31-b532-b815b3b58626-os-release\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.680320 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.679377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e48d716c-b5ee-4a31-b532-b815b3b58626-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.680938 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.680860    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13fb85dc-3804-4c03-9735-fe08cd156f0d-etc-tuned\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.681599 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.681563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13fb85dc-3804-4c03-9735-fe08cd156f0d-tmp\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.687044 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.686830    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:55.687044 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.686860    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:55.687044 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.686876    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fnmdt for pod openshift-network-diagnostics/network-check-target-fjdnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:55.687044 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:55.686959    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt podName:6b52971f-2c3f-4986-a372-2c8d98ea58da nodeName:}" failed. No retries permitted until 2026-07-10 18:39:56.186936654 +0000 UTC m=+3.141224722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fnmdt" (UniqueName: "kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt") pod "network-check-target-fjdnw" (UID: "6b52971f-2c3f-4986-a372-2c8d98ea58da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:55.688360 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.688333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqw2\" (UniqueName: \"kubernetes.io/projected/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-kube-api-access-grqw2\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:55.688763 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.688710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcnzj\" (UniqueName: \"kubernetes.io/projected/e8345347-19c1-46c8-8524-b906e0a9fbbf-kube-api-access-kcnzj\") pod \"aws-ebs-csi-driver-node-ctln2\" (UID: \"e8345347-19c1-46c8-8524-b906e0a9fbbf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.689069 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.689048    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrc5\" (UniqueName: \"kubernetes.io/projected/0132a4d3-2442-4059-bdb6-28f3b2fbc18d-kube-api-access-scrc5\") pod \"multus-kkdrl\" (UID: \"0132a4d3-2442-4059-bdb6-28f3b2fbc18d\") " pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.689525 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.689496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqkz\" (UniqueName: \"kubernetes.io/projected/13fb85dc-3804-4c03-9735-fe08cd156f0d-kube-api-access-fhqkz\") pod \"tuned-z58qt\" (UID: \"13fb85dc-3804-4c03-9735-fe08cd156f0d\") " pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:55.690813 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.690784    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tn2q\" (UniqueName: \"kubernetes.io/projected/3f921aa9-6e25-4f9c-a45c-3eb718f655a9-kube-api-access-8tn2q\") pod \"node-ca-8cvcp\" (UID: \"3f921aa9-6e25-4f9c-a45c-3eb718f655a9\") " pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.691537 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.691511    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpj2\" (UniqueName: \"kubernetes.io/projected/d968d33f-746c-4f85-8118-64bf2e41a3ee-kube-api-access-4lpj2\") pod \"iptables-alerter-f267f\" (UID: \"d968d33f-746c-4f85-8118-64bf2e41a3ee\") " pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.691537 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.691525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm29s\" (UniqueName: \"kubernetes.io/projected/e48d716c-b5ee-4a31-b532-b815b3b58626-kube-api-access-dm29s\") pod \"multus-additional-cni-plugins-2gkfp\" (UID: \"e48d716c-b5ee-4a31-b532-b815b3b58626\") " pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.764488 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.764443    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hhb4q"
Jul 10 18:39:55.783273 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.783236    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:39:55.794525 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.794490    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:39:55.800574 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.800548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gkfp"
Jul 10 18:39:55.810621 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.810589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2"
Jul 10 18:39:55.821811 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.821781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kkdrl"
Jul 10 18:39:55.830160 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.830119    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8cvcp"
Jul 10 18:39:55.838353 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.838323    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f267f"
Jul 10 18:39:55.847421 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:55.847392    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z58qt"
Jul 10 18:39:56.083497 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:56.083403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:56.083676 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.083603    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:56.083738 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.083686    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret podName:3c91b6f5-d69d-47d5-9782-18d0a40d6f4e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:57.083664795 +0000 UTC m=+4.037952842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret") pod "global-pull-secret-syncer-krjlw" (UID: "3c91b6f5-d69d-47d5-9782-18d0a40d6f4e") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:56.184788 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:56.184742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:56.184995 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.184934    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:56.185055 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.185014    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:57.184996886 +0000 UTC m=+4.139284950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:56.285977 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:56.285940    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:56.286179 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.286088    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:56.286179 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.286108    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:56.286179 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.286122    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fnmdt for pod openshift-network-diagnostics/network-check-target-fjdnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:56.286300 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.286198    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt podName:6b52971f-2c3f-4986-a372-2c8d98ea58da nodeName:}" failed. No retries permitted until 2026-07-10 18:39:57.286167536 +0000 UTC m=+4.240455585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnmdt" (UniqueName: "kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt") pod "network-check-target-fjdnw" (UID: "6b52971f-2c3f-4986-a372-2c8d98ea58da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:56.535823 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:56.535732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:56.536369 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:56.535927    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:39:56.549279 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:56.549215    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:34:55 +0000 UTC" deadline="2027-12-26 10:32:37.458763411 +0000 UTC"
Jul 10 18:39:56.549279 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:56.549270    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12807h52m40.909497566s"
Jul 10 18:39:56.709220 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.709187    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fb85dc_3804_4c03_9735_fe08cd156f0d.slice/crio-66cfd00d65169afe0a77ac16beb2cc94ea1152b8ce058bed449192e4ef98ddc3 WatchSource:0}: Error finding container 66cfd00d65169afe0a77ac16beb2cc94ea1152b8ce058bed449192e4ef98ddc3: Status 404 returned error can't find the container with id 66cfd00d65169afe0a77ac16beb2cc94ea1152b8ce058bed449192e4ef98ddc3
Jul 10 18:39:56.732302 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.732269    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd4ef1f_f28c_4979_bc40_20da7f09a697.slice/crio-e68d786505a37fb89acd4088da792624fd89ad567deef3b7715fdcaf6d513427 WatchSource:0}: Error finding container e68d786505a37fb89acd4088da792624fd89ad567deef3b7715fdcaf6d513427: Status 404 returned error can't find the container with id e68d786505a37fb89acd4088da792624fd89ad567deef3b7715fdcaf6d513427
Jul 10 18:39:56.732840 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.732812    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee32311a_4aa2_4168_bffd_f355e8dae244.slice/crio-455a8b46b88db7a00ff077f5eb1ef022072b2c180360de9d23cc4c9f47167d7d WatchSource:0}: Error finding container 455a8b46b88db7a00ff077f5eb1ef022072b2c180360de9d23cc4c9f47167d7d: Status 404 returned error can't find the container with id 455a8b46b88db7a00ff077f5eb1ef022072b2c180360de9d23cc4c9f47167d7d
Jul 10 18:39:56.734498 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.734370    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48d716c_b5ee_4a31_b532_b815b3b58626.slice/crio-74d597c8f1e361b933685004be17aa2db43c44bd7065327026c48fe27a748a21 WatchSource:0}: Error finding container 74d597c8f1e361b933685004be17aa2db43c44bd7065327026c48fe27a748a21: Status 404 returned error can't find the container with id 74d597c8f1e361b933685004be17aa2db43c44bd7065327026c48fe27a748a21
Jul 10 18:39:56.736194 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.736172    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8345347_19c1_46c8_8524_b906e0a9fbbf.slice/crio-4260c46d3b8e87f5ece8737b6263a234edd6e4841e5d5a8f5443290b7b78a96f WatchSource:0}: Error finding container 4260c46d3b8e87f5ece8737b6263a234edd6e4841e5d5a8f5443290b7b78a96f: Status 404 returned error can't find the container with id 4260c46d3b8e87f5ece8737b6263a234edd6e4841e5d5a8f5443290b7b78a96f
Jul 10 18:39:56.742181 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.741938    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd968d33f_746c_4f85_8118_64bf2e41a3ee.slice/crio-efa21b66efaf7c6d125aad72d14bcd0f9caa9de43f6746f54f6c29567361ed34 WatchSource:0}: Error finding container efa21b66efaf7c6d125aad72d14bcd0f9caa9de43f6746f54f6c29567361ed34: Status 404 returned error can't find the container with id efa21b66efaf7c6d125aad72d14bcd0f9caa9de43f6746f54f6c29567361ed34
Jul 10 18:39:56.742938 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.742914    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f921aa9_6e25_4f9c_a45c_3eb718f655a9.slice/crio-8cf36e4690b4e21b0409a0a97daf29228a0dc89eebe75221200e62f801c28e48 WatchSource:0}: Error finding container 8cf36e4690b4e21b0409a0a97daf29228a0dc89eebe75221200e62f801c28e48: Status 404 returned error can't find the container with id 8cf36e4690b4e21b0409a0a97daf29228a0dc89eebe75221200e62f801c28e48
Jul 10 18:39:56.744416 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.744321    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0132a4d3_2442_4059_bdb6_28f3b2fbc18d.slice/crio-330377b5b88882eb653e1d1393291c1234b35f59d8f5dc2b95188f63e3389a05 WatchSource:0}: Error finding container 330377b5b88882eb653e1d1393291c1234b35f59d8f5dc2b95188f63e3389a05: Status 404 returned error can't find the container with id 330377b5b88882eb653e1d1393291c1234b35f59d8f5dc2b95188f63e3389a05
Jul 10 18:39:56.745454 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:39:56.745426    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3dac8e_46b2_4462_9b80_f1ddd3732eed.slice/crio-f9bbb526790339c9fd930c4c1de9dc84d4932d43bb4bcc6f312c2452ea504fb7 WatchSource:0}: Error finding container f9bbb526790339c9fd930c4c1de9dc84d4932d43bb4bcc6f312c2452ea504fb7: Status 404 returned error can't find the container with id f9bbb526790339c9fd930c4c1de9dc84d4932d43bb4bcc6f312c2452ea504fb7
Jul 10 18:39:57.092840 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.092439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:57.092840 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.092637    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:57.093063 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.092893    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret podName:3c91b6f5-d69d-47d5-9782-18d0a40d6f4e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:59.092866929 +0000 UTC m=+6.047154984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret") pod "global-pull-secret-syncer-krjlw" (UID: "3c91b6f5-d69d-47d5-9782-18d0a40d6f4e") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:57.194116 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.194076    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:57.194314 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.194277    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:57.194408 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.194347    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:59.194326896 +0000 UTC m=+6.148614948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:57.295664 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.294909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:57.295664 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.295123    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:57.295664 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.295142    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:57.295664 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.295157    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fnmdt for pod openshift-network-diagnostics/network-check-target-fjdnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:57.295664 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.295241    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt podName:6b52971f-2c3f-4986-a372-2c8d98ea58da nodeName:}" failed. No retries permitted until 2026-07-10 18:39:59.295220738 +0000 UTC m=+6.249508811 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnmdt" (UniqueName: "kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt") pod "network-check-target-fjdnw" (UID: "6b52971f-2c3f-4986-a372-2c8d98ea58da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:57.538023 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.537989    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:57.538716 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.538145    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:39:57.538716 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.538700    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:57.538836 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:57.538814    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:39:57.550150 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.550049    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:34:55 +0000 UTC" deadline="2028-03-28 11:43:20.271782284 +0000 UTC"
Jul 10 18:39:57.550150 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.550107    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15041h3m22.721678974s"
Jul 10 18:39:57.570768 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.570654    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kkdrl" event={"ID":"0132a4d3-2442-4059-bdb6-28f3b2fbc18d","Type":"ContainerStarted","Data":"330377b5b88882eb653e1d1393291c1234b35f59d8f5dc2b95188f63e3389a05"}
Jul 10 18:39:57.576463 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.576377    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2" event={"ID":"e8345347-19c1-46c8-8524-b906e0a9fbbf","Type":"ContainerStarted","Data":"4260c46d3b8e87f5ece8737b6263a234edd6e4841e5d5a8f5443290b7b78a96f"}
Jul 10 18:39:57.590573 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.590445    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerStarted","Data":"74d597c8f1e361b933685004be17aa2db43c44bd7065327026c48fe27a748a21"}
Jul 10 18:39:57.605783 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.605730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hhb4q" event={"ID":"ee32311a-4aa2-4168-bffd-f355e8dae244","Type":"ContainerStarted","Data":"455a8b46b88db7a00ff077f5eb1ef022072b2c180360de9d23cc4c9f47167d7d"}
Jul 10 18:39:57.622698 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.622656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal" event={"ID":"fc0950c7cf4a4407bce2f7a337f490f9","Type":"ContainerStarted","Data":"a8fa5eba07121d8bf082147b85840f9543bc5f3640826e5ff2ebe2e1ad3d2ded"}
Jul 10 18:39:57.639966 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.639922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"f9bbb526790339c9fd930c4c1de9dc84d4932d43bb4bcc6f312c2452ea504fb7"}
Jul 10 18:39:57.644432 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.644366    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-57.ec2.internal" podStartSLOduration=2.644344235 podStartE2EDuration="2.644344235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:57.643475783 +0000 UTC m=+4.597763855" watchObservedRunningTime="2026-07-10 18:39:57.644344235 +0000 UTC m=+4.598632308"
Jul 10 18:39:57.649928 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.649847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8cvcp" event={"ID":"3f921aa9-6e25-4f9c-a45c-3eb718f655a9","Type":"ContainerStarted","Data":"8cf36e4690b4e21b0409a0a97daf29228a0dc89eebe75221200e62f801c28e48"}
Jul 10 18:39:57.652824 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.652786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f267f" event={"ID":"d968d33f-746c-4f85-8118-64bf2e41a3ee","Type":"ContainerStarted","Data":"efa21b66efaf7c6d125aad72d14bcd0f9caa9de43f6746f54f6c29567361ed34"}
Jul 10 18:39:57.655315 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.655211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zw7sz" event={"ID":"dfd4ef1f-f28c-4979-bc40-20da7f09a697","Type":"ContainerStarted","Data":"e68d786505a37fb89acd4088da792624fd89ad567deef3b7715fdcaf6d513427"}
Jul 10 18:39:57.657639 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.657610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z58qt" event={"ID":"13fb85dc-3804-4c03-9735-fe08cd156f0d","Type":"ContainerStarted","Data":"66cfd00d65169afe0a77ac16beb2cc94ea1152b8ce058bed449192e4ef98ddc3"}
Jul 10 18:39:57.664717 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.664316    2561 generic.go:356] "Generic (PLEG): container finished" podID="40921e7d9f497f62d3e2dfd9e8ed1bc8" containerID="2d4807c0b10a9daa81fe34a9a6ab46d1b99ab936225ae7a05a774ed2214522ab" exitCode=0
Jul 10 18:39:57.664717 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:57.664367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal" event={"ID":"40921e7d9f497f62d3e2dfd9e8ed1bc8","Type":"ContainerDied","Data":"2d4807c0b10a9daa81fe34a9a6ab46d1b99ab936225ae7a05a774ed2214522ab"}
Jul 10 18:39:58.535776 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:58.535726    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:58.535975 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:58.535917    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:39:58.715853 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:58.715385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal" event={"ID":"40921e7d9f497f62d3e2dfd9e8ed1bc8","Type":"ContainerStarted","Data":"35df578c2a0ad0c215845465bde47d61cdb756b9eb481e5b607f5366ecaa025e"}
Jul 10 18:39:58.733260 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:58.732487    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-57.ec2.internal" podStartSLOduration=3.732459842 podStartE2EDuration="3.732459842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:58.731651927 +0000 UTC m=+5.685939998" watchObservedRunningTime="2026-07-10 18:39:58.732459842 +0000 UTC m=+5.686747916"
Jul 10 18:39:59.108981 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:59.108938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:59.109190 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.109111    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:59.109190 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.109179    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret podName:3c91b6f5-d69d-47d5-9782-18d0a40d6f4e nodeName:}" failed. No retries permitted until 2026-07-10 18:40:03.109159259 +0000 UTC m=+10.063447311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret") pod "global-pull-secret-syncer-krjlw" (UID: "3c91b6f5-d69d-47d5-9782-18d0a40d6f4e") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:59.209787 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:59.209741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:39:59.209999 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.209923    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:59.210069 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.210005    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:03.209983266 +0000 UTC m=+10.164271328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:59.311148 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:59.311106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:59.311395 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.311373    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:59.311491 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.311403    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:59.311491 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.311417    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fnmdt for pod openshift-network-diagnostics/network-check-target-fjdnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:59.311491 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.311477    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt podName:6b52971f-2c3f-4986-a372-2c8d98ea58da nodeName:}" failed. No retries permitted until 2026-07-10 18:40:03.31145635 +0000 UTC m=+10.265744399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnmdt" (UniqueName: "kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt") pod "network-check-target-fjdnw" (UID: "6b52971f-2c3f-4986-a372-2c8d98ea58da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:59.536018 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:59.535923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:39:59.536228 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.536081    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:39:59.537235 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:39:59.537198    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:39:59.537393 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:39:59.537333    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:00.535799 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:00.535463    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:00.535799 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:00.535617    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:01.535825 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:01.535793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:01.536285 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:01.535932    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:01.536285 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:01.536042    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:01.536285 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:01.536180    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:02.535513 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:02.535478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:02.535689 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:02.535619    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:03.149190 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:03.149048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:03.149664 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.149197    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:03.149664 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.149266    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret podName:3c91b6f5-d69d-47d5-9782-18d0a40d6f4e nodeName:}" failed. No retries permitted until 2026-07-10 18:40:11.149247116 +0000 UTC m=+18.103535160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret") pod "global-pull-secret-syncer-krjlw" (UID: "3c91b6f5-d69d-47d5-9782-18d0a40d6f4e") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:03.250132 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:03.250085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:03.250328 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.250288    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:03.250415 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.250368    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:11.250347574 +0000 UTC m=+18.204635625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:03.351637 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:03.351545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:03.351807 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.351766    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:03.351807 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.351791    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:03.351807 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.351803    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fnmdt for pod openshift-network-diagnostics/network-check-target-fjdnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:03.351963 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.351870    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt podName:6b52971f-2c3f-4986-a372-2c8d98ea58da nodeName:}" failed. No retries permitted until 2026-07-10 18:40:11.351850392 +0000 UTC m=+18.306138452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnmdt" (UniqueName: "kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt") pod "network-check-target-fjdnw" (UID: "6b52971f-2c3f-4986-a372-2c8d98ea58da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:03.538604 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:03.538453    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:03.538604 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:03.538478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:03.538604 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.538589    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:03.539939 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:03.539883    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:04.535828 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:04.535781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:04.536307 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:04.535921    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:05.535547 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:05.535500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:05.535722 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:05.535646    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:05.537282 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:05.537238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:05.537720 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:05.537374    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:06.535927 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:06.535890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:06.536126 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:06.536032    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:07.535534 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:07.535494    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:07.535534 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:07.535517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:07.536063 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:07.535630    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:07.536063 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:07.535721    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:08.535874 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:08.535834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:08.536338 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:08.535984    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:09.535411 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:09.535367    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:09.535597 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:09.535501    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:09.535597 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:09.535552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:09.535715 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:09.535649    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:10.535776 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:10.535740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:10.536288 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:10.535877    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:11.215130 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:11.215093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:11.215307 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.215215    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:11.215307 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.215270    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret podName:3c91b6f5-d69d-47d5-9782-18d0a40d6f4e nodeName:}" failed. No retries permitted until 2026-07-10 18:40:27.215254616 +0000 UTC m=+34.169542661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret") pod "global-pull-secret-syncer-krjlw" (UID: "3c91b6f5-d69d-47d5-9782-18d0a40d6f4e") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:11.316321 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:11.316280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:11.316520 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.316442    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:11.316520 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.316519    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:27.316497957 +0000 UTC m=+34.270786002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:11.417154 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:11.417124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:11.417367 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.417267    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:11.417367 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.417285    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:11.417367 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.417294    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fnmdt for pod openshift-network-diagnostics/network-check-target-fjdnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:11.417367 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.417342    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt podName:6b52971f-2c3f-4986-a372-2c8d98ea58da nodeName:}" failed. No retries permitted until 2026-07-10 18:40:27.417328407 +0000 UTC m=+34.371616451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnmdt" (UniqueName: "kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt") pod "network-check-target-fjdnw" (UID: "6b52971f-2c3f-4986-a372-2c8d98ea58da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:11.535877 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:11.535834    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:11.536316 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.535962    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:11.536316 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:11.536012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:11.536316 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:11.536095    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:12.535164 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:12.535121    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:12.535386 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:12.535254    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:13.536808 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:13.536766    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:13.537356 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:13.536890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:13.537356 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:13.536976    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:13.537356 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:13.536883    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:14.536037 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:14.536001    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:14.536204 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:14.536128    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:15.536218 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:15.536184    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:15.536712 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:15.536192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:15.536712 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:15.536330    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:15.536712 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:15.536365    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:16.535729 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.535288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:16.535905 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:16.535864    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:16.768126 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.768086    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2" event={"ID":"e8345347-19c1-46c8-8524-b906e0a9fbbf","Type":"ContainerStarted","Data":"67d4fcb658d13632f1335d42e083c6426319b6863856cde9bbb3c51687b9236b"}
Jul 10 18:40:16.771397 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.771365    2561 generic.go:356] "Generic (PLEG): container finished" podID="e48d716c-b5ee-4a31-b532-b815b3b58626" containerID="fd8c322da8057e5fe270e51750e84a69165861d3cc2a77dbebcba289a29ec196" exitCode=0
Jul 10 18:40:16.771550 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.771421    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerDied","Data":"fd8c322da8057e5fe270e51750e84a69165861d3cc2a77dbebcba289a29ec196"}
Jul 10 18:40:16.773117 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.773084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hhb4q" event={"ID":"ee32311a-4aa2-4168-bffd-f355e8dae244","Type":"ContainerStarted","Data":"60023a648fefbcd2e7866965e40643b7c86e7e343a8478ae61db5df4a134120f"}
Jul 10 18:40:16.776083 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.776058    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"fe0120da271735ac9144032161b4c5d2d27f5ab2d9d5fb965ec7b2213c3ba597"}
Jul 10 18:40:16.776213 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.776129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"dced436108c96a814529af167c77fd65d898002e67ace941128b2531fe26551b"}
Jul 10 18:40:16.776213 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.776194    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"867db827ae1dfec7fc3f04e69f2231de0623511e88ac46ec5b3f3d90b6d95f1f"}
Jul 10 18:40:16.776335 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.776244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"7895a42325762f813ca97a257e41e786d04a547fa26e30c392b2be11dbc36527"}
Jul 10 18:40:16.776335 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.776277    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"aa76e350fcc1f82b4dc6b28e7101b3336add520df893e7e3d18549210bcef38c"}
Jul 10 18:40:16.776335 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.776313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"36cebe57c3a4a98321f2a8acc89010b0e2431fe57180fef7869a4f13d360b62c"}
Jul 10 18:40:16.777484 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.777458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8cvcp" event={"ID":"3f921aa9-6e25-4f9c-a45c-3eb718f655a9","Type":"ContainerStarted","Data":"70f37c0e97cdcd83f335f28d45a2be76a20df59a331d4077dc9017ae1e92d71e"}
Jul 10 18:40:16.778869 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.778836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zw7sz" event={"ID":"dfd4ef1f-f28c-4979-bc40-20da7f09a697","Type":"ContainerStarted","Data":"3cf00b9a0cb281bc3060a9b67fbd48eecdb77a972dd7832d8895ad60e1546e3f"}
Jul 10 18:40:16.780295 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.780274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z58qt" event={"ID":"13fb85dc-3804-4c03-9735-fe08cd156f0d","Type":"ContainerStarted","Data":"ec242b8274267522c73c7b6d9c8566d32d5cb86514b4c2364555e2208400c871"}
Jul 10 18:40:16.781570 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.781549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kkdrl" event={"ID":"0132a4d3-2442-4059-bdb6-28f3b2fbc18d","Type":"ContainerStarted","Data":"042c1590a44255efd2687c9fa1f3ab5d6d61924cb4601f15406f76d82e39064e"}
Jul 10 18:40:16.820889 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.820830    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z58qt" podStartSLOduration=3.90191677 podStartE2EDuration="22.82081216s" totalImagesPullingTime="18.91889539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:16.820776555 +0000 UTC m=+23.775064623" watchObservedRunningTime="2026-07-10 18:40:16.82081216 +0000 UTC m=+23.775100228"
Jul 10 18:40:16.837014 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.836953    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zw7sz" podStartSLOduration=8.116701754 podStartE2EDuration="22.836934425s" totalImagesPullingTime="14.720232671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:16.836457616 +0000 UTC m=+23.790745684" watchObservedRunningTime="2026-07-10 18:40:16.836934425 +0000 UTC m=+23.791222493"
Jul 10 18:40:16.863010 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.862960    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kkdrl" podStartSLOduration=3.941744719 podStartE2EDuration="22.862944856s" totalImagesPullingTime="18.921200137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:16.861746725 +0000 UTC m=+23.816034793" watchObservedRunningTime="2026-07-10 18:40:16.862944856 +0000 UTC m=+23.817232920"
Jul 10 18:40:16.888087 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.888037    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8cvcp" podStartSLOduration=8.178697947 podStartE2EDuration="22.888022537s" totalImagesPullingTime="14.70932459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:16.887827413 +0000 UTC m=+23.842115482" watchObservedRunningTime="2026-07-10 18:40:16.888022537 +0000 UTC m=+23.842310669"
Jul 10 18:40:16.904454 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:16.904371    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hhb4q" podStartSLOduration=4.01625806 podStartE2EDuration="22.904352074s" totalImagesPullingTime="18.888094014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:16.903629382 +0000 UTC m=+23.857917453" watchObservedRunningTime="2026-07-10 18:40:16.904352074 +0000 UTC m=+23.858640140"
Jul 10 18:40:17.233068 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.233027    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:40:17.532650 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.532496    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:40:17.233061766Z","UUID":"17a786b8-ef84-43c3-be78-3072f7d64b23","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:40:17.533891 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.533871    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:40:17.534014 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.533910    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:40:17.536198 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.536174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:17.536334 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:17.536281    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:17.536383 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.536333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:17.536433 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:17.536414    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:17.785232 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.785133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f267f" event={"ID":"d968d33f-746c-4f85-8118-64bf2e41a3ee","Type":"ContainerStarted","Data":"9753b25bd98a54326d18e8cd3de48d4f4ffb9a841dcf85d79109d14f764c9b4b"}
Jul 10 18:40:17.786978 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.786948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2" event={"ID":"e8345347-19c1-46c8-8524-b906e0a9fbbf","Type":"ContainerStarted","Data":"d98f628c2b393dc85af44404d967181d25fbbb65e758ad0f162c239af0007238"}
Jul 10 18:40:17.799637 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:17.799581    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f267f" podStartSLOduration=4.921010222 podStartE2EDuration="23.799564517s" totalImagesPullingTime="18.878554295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:17.799370924 +0000 UTC m=+24.753658991" watchObservedRunningTime="2026-07-10 18:40:17.799564517 +0000 UTC m=+24.753852584"
Jul 10 18:40:18.535911 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:18.535878    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:18.536097 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:18.536021    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:18.792468 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:18.792366    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"9e51844249ccdaeacf6f36ef776d0ef9831d5194633824d863c668f6ba5b85ed"}
Jul 10 18:40:18.796497 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:18.796460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2" event={"ID":"e8345347-19c1-46c8-8524-b906e0a9fbbf","Type":"ContainerStarted","Data":"d6e3ce3116379f4745c264b69925d63a192bf339870c4fcab37eb06abfc7761e"}
Jul 10 18:40:18.815488 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:18.815437    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ctln2" podStartSLOduration=3.436122611 podStartE2EDuration="24.815419611s" totalImagesPullingTime="21.379297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:18.814270071 +0000 UTC m=+25.768558137" watchObservedRunningTime="2026-07-10 18:40:18.815419611 +0000 UTC m=+25.769707675"
Jul 10 18:40:19.535892 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:19.535854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:19.536094 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:19.535987    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:19.536094 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:19.536043    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:19.536233 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:19.536214    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:20.111504 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:20.111307    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:40:20.111948 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:20.111931    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:40:20.535653 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:20.535552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:20.535828 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:20.535701    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:20.808728 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:20.808693    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" event={"ID":"cc3dac8e-46b2-4462-9b80-f1ddd3732eed","Type":"ContainerStarted","Data":"84dafdaa79cf63aa985137d5bc806c33db01cc83cfc91dc8e7ef5475d812b222"}
Jul 10 18:40:20.808922 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:20.808891    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:40:20.809445 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:20.809421    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zw7sz"
Jul 10 18:40:20.842593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:20.842514    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nltht" podStartSLOduration=7.874039206 podStartE2EDuration="26.842496005s" totalImagesPullingTime="18.968456799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:20.840406452 +0000 UTC m=+27.794694520" watchObservedRunningTime="2026-07-10 18:40:20.842496005 +0000 UTC m=+27.796784169"
Jul 10 18:40:21.535795 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:21.535753    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:21.536808 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:21.535757    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:21.536808 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:21.535913    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:21.536808 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:21.535985    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:21.811205 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:21.811102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:40:21.811334 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:21.811218    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:40:21.811334 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:21.811262    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:40:21.829599 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:21.829566    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:40:21.830400 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:21.830375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:40:22.536334 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:22.536105    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:22.536808 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:22.536483    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:22.686178 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:22.686112    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krjlw"]
Jul 10 18:40:22.686361 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:22.686278    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:22.686412 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:22.686396    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:22.689324 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:22.689290    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjdnw"]
Jul 10 18:40:22.689483 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:22.689408    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:22.689532 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:22.689513    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:22.696261 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:22.696226    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sbhh"]
Jul 10 18:40:22.812647 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:22.812571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:22.812799 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:22.812767    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:24.535240 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:24.535201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:24.535686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:24.535201    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:24.535686 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:24.535347    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:24.535686 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:24.535220    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:24.535686 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:24.535384    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:24.535686 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:24.535461    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:24.819295 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:24.819266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerStarted","Data":"edb6823d82842d371fc2034cb2584e7c3c8581ba212fa1bcc6c202ac34ce0b51"}
Jul 10 18:40:25.822874 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:25.822838    2561 generic.go:356] "Generic (PLEG): container finished" podID="e48d716c-b5ee-4a31-b532-b815b3b58626" containerID="edb6823d82842d371fc2034cb2584e7c3c8581ba212fa1bcc6c202ac34ce0b51" exitCode=0
Jul 10 18:40:25.823416 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:25.822894    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerDied","Data":"edb6823d82842d371fc2034cb2584e7c3c8581ba212fa1bcc6c202ac34ce0b51"}
Jul 10 18:40:26.535625 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:26.535589    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:26.535785 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:26.535587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:26.535785 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:26.535737    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sbhh" podUID="f6e8e427-7654-41f4-a0cf-8b4e07fc6381"
Jul 10 18:40:26.535874 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:26.535848    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjdnw" podUID="6b52971f-2c3f-4986-a372-2c8d98ea58da"
Jul 10 18:40:26.535912 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:26.535587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:26.535963 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:26.535945    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krjlw" podUID="3c91b6f5-d69d-47d5-9782-18d0a40d6f4e"
Jul 10 18:40:27.231755 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:27.231565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:27.231755 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.231712    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:27.232182 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.231815    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret podName:3c91b6f5-d69d-47d5-9782-18d0a40d6f4e nodeName:}" failed. No retries permitted until 2026-07-10 18:40:59.231798569 +0000 UTC m=+66.186086615 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret") pod "global-pull-secret-syncer-krjlw" (UID: "3c91b6f5-d69d-47d5-9782-18d0a40d6f4e") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:40:27.332285 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:27.332194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:27.332425 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.332302    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:27.332425 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.332355    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:59.332340621 +0000 UTC m=+66.286628665 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:40:27.432843 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:27.432802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:27.432981 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.432938    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:40:27.432981 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.432951    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:40:27.432981 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.432961    2561 projected.go:196] Error preparing data for projected volume kube-api-access-fnmdt for pod openshift-network-diagnostics/network-check-target-fjdnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:27.433098 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:27.433007    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt podName:6b52971f-2c3f-4986-a372-2c8d98ea58da nodeName:}" failed. No retries permitted until 2026-07-10 18:40:59.43299518 +0000 UTC m=+66.387283225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fnmdt" (UniqueName: "kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt") pod "network-check-target-fjdnw" (UID: "6b52971f-2c3f-4986-a372-2c8d98ea58da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:40:27.829622 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:27.829538    2561 generic.go:356] "Generic (PLEG): container finished" podID="e48d716c-b5ee-4a31-b532-b815b3b58626" containerID="b4d7db32f2b326ca0257659498a73989688dd46e5ac63319c83a43130696e5ab" exitCode=0
Jul 10 18:40:27.829622 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:27.829597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerDied","Data":"b4d7db32f2b326ca0257659498a73989688dd46e5ac63319c83a43130696e5ab"}
Jul 10 18:40:28.289308 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.289282    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-57.ec2.internal" event="NodeReady"
Jul 10 18:40:28.289735 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.289480    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:40:28.323386 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.323358    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76c4744ff8-jn9h4"]
Jul 10 18:40:28.336612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.336581    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k9d5q"]
Jul 10 18:40:28.336813 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.336691    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.339654 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.339615    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 10 18:40:28.339654 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.339629    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-znrdn\""
Jul 10 18:40:28.339832 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.339694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 10 18:40:28.339832 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.339621    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 10 18:40:28.348624 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.348595    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 10 18:40:28.356841 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.356811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c4744ff8-jn9h4"]
Jul 10 18:40:28.356953 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.356858    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rdwq2"]
Jul 10 18:40:28.357001 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.356982    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.359875 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.359851    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:40:28.360013 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.360000    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:40:28.360165 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.360130    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-stn59\""
Jul 10 18:40:28.372837 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.372811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9d5q"]
Jul 10 18:40:28.372955 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.372858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rdwq2"]
Jul 10 18:40:28.372955 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.372942    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.375834 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.375806    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:40:28.375834 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.375817    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:40:28.375834 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.375806    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pkb85\""
Jul 10 18:40:28.376054 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.375941    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:40:28.441671 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8c92\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-kube-api-access-v8c92\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.441671 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.441864 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-tmp-dir\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.441864 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/612b8f12-28aa-444e-beb3-5700ebd338f3-ca-trust-extracted\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.441864 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-image-registry-private-configuration\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.441864 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-certificates\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.441864 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-config-volume\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.441864 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.441864 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441857    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-bound-sa-token\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.442063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-trusted-ca\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.442063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hz6k\" (UniqueName: \"kubernetes.io/projected/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-kube-api-access-8hz6k\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.442063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.441957    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-installation-pull-secrets\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.535573 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.535526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:28.535573 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.535563    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:28.535794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.535526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:28.541394 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.541370    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:40:28.542293 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-bound-sa-token\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542371 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-trusted-ca\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542371 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hz6k\" (UniqueName: \"kubernetes.io/projected/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-kube-api-access-8hz6k\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.542371 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542366    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-installation-pull-secrets\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542524 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62skg\" (UniqueName: \"kubernetes.io/projected/01dc8b49-c2dd-42af-a45d-9dac0564b514-kube-api-access-62skg\") pod \"ingress-canary-rdwq2\" (UID: \"01dc8b49-c2dd-42af-a45d-9dac0564b514\") " pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.542524 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8c92\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-kube-api-access-v8c92\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542524 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542524 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-tmp-dir\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.542713 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/612b8f12-28aa-444e-beb3-5700ebd338f3-ca-trust-extracted\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542713 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-image-registry-private-configuration\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542713 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-certificates\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.542713 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01dc8b49-c2dd-42af-a45d-9dac0564b514-cert\") pod \"ingress-canary-rdwq2\" (UID: \"01dc8b49-c2dd-42af-a45d-9dac0564b514\") " pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.542713 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-config-volume\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.542713 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.542709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.542992 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:28.542821    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:28.542992 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:28.542918    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls podName:d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:29.042900016 +0000 UTC m=+35.997188061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls") pod "dns-default-k9d5q" (UID: "d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:28.543253 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:28.543234    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:28.543253 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:28.543252    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c4744ff8-jn9h4: secret "image-registry-tls" not found
Jul 10 18:40:28.543484 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:28.543300    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls podName:612b8f12-28aa-444e-beb3-5700ebd338f3 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:29.043282657 +0000 UTC m=+35.997570706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls") pod "image-registry-76c4744ff8-jn9h4" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3") : secret "image-registry-tls" not found
Jul 10 18:40:28.543484 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.543370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-trusted-ca\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.543602 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.543570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-tmp-dir\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.543602 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.543583    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-config-volume\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.543742 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.543720    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/612b8f12-28aa-444e-beb3-5700ebd338f3-ca-trust-extracted\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.543841 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.543820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-certificates\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.544695 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.544663    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v9r2k\""
Jul 10 18:40:28.544788 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.544670    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:40:28.546800 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.546780    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p7x9n\""
Jul 10 18:40:28.546915 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.546897    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:40:28.546973 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.546915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:40:28.547557 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.547537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-image-registry-private-configuration\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.547632 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.547571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-installation-pull-secrets\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.564728 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.564701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-bound-sa-token\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.564920 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.564898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8c92\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-kube-api-access-v8c92\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:28.568757 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.568730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hz6k\" (UniqueName: \"kubernetes.io/projected/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-kube-api-access-8hz6k\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:28.643628 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.643606    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01dc8b49-c2dd-42af-a45d-9dac0564b514-cert\") pod \"ingress-canary-rdwq2\" (UID: \"01dc8b49-c2dd-42af-a45d-9dac0564b514\") " pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.643690 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.643665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62skg\" (UniqueName: \"kubernetes.io/projected/01dc8b49-c2dd-42af-a45d-9dac0564b514-kube-api-access-62skg\") pod \"ingress-canary-rdwq2\" (UID: \"01dc8b49-c2dd-42af-a45d-9dac0564b514\") " pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.646025 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.646006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01dc8b49-c2dd-42af-a45d-9dac0564b514-cert\") pod \"ingress-canary-rdwq2\" (UID: \"01dc8b49-c2dd-42af-a45d-9dac0564b514\") " pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.654670 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.654645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62skg\" (UniqueName: \"kubernetes.io/projected/01dc8b49-c2dd-42af-a45d-9dac0564b514-kube-api-access-62skg\") pod \"ingress-canary-rdwq2\" (UID: \"01dc8b49-c2dd-42af-a45d-9dac0564b514\") " pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.682157 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.682014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rdwq2"
Jul 10 18:40:28.835062 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.835032    2561 generic.go:356] "Generic (PLEG): container finished" podID="e48d716c-b5ee-4a31-b532-b815b3b58626" containerID="c0cc0ad5f351a7998488beb800058af9fa0d3e106f8e2516d77f8f2dbad2a4fe" exitCode=0
Jul 10 18:40:28.835229 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.835105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerDied","Data":"c0cc0ad5f351a7998488beb800058af9fa0d3e106f8e2516d77f8f2dbad2a4fe"}
Jul 10 18:40:28.846006 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:28.845978    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rdwq2"]
Jul 10 18:40:28.850911 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:28.850727    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dc8b49_c2dd_42af_a45d_9dac0564b514.slice/crio-e24f07da440885e3d6b672e8437dd35e7621d5a02f1673aee9854755a62f96de WatchSource:0}: Error finding container e24f07da440885e3d6b672e8437dd35e7621d5a02f1673aee9854755a62f96de: Status 404 returned error can't find the container with id e24f07da440885e3d6b672e8437dd35e7621d5a02f1673aee9854755a62f96de
Jul 10 18:40:29.047893 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:29.047793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:29.047893 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:29.047873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:29.048140 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:29.047950    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:29.048140 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:29.047974    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c4744ff8-jn9h4: secret "image-registry-tls" not found
Jul 10 18:40:29.048140 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:29.048008    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:29.048140 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:29.048032    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls podName:612b8f12-28aa-444e-beb3-5700ebd338f3 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:30.048013735 +0000 UTC m=+37.002301801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls") pod "image-registry-76c4744ff8-jn9h4" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3") : secret "image-registry-tls" not found
Jul 10 18:40:29.048140 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:29.048096    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls podName:d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:30.048079439 +0000 UTC m=+37.002367490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls") pod "dns-default-k9d5q" (UID: "d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:29.622449 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:29.622402    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:29.720466 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:29.720369    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:29.839985 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:29.839923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rdwq2" event={"ID":"01dc8b49-c2dd-42af-a45d-9dac0564b514","Type":"ContainerStarted","Data":"e24f07da440885e3d6b672e8437dd35e7621d5a02f1673aee9854755a62f96de"}
Jul 10 18:40:30.057309 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:30.057216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:30.057477 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:30.057417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:30.057477 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:30.057432    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:30.057595 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:30.057533    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls podName:d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:32.057483961 +0000 UTC m=+39.011772015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls") pod "dns-default-k9d5q" (UID: "d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:30.057595 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:30.057589    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:30.057701 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:30.057605    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c4744ff8-jn9h4: secret "image-registry-tls" not found
Jul 10 18:40:30.057701 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:30.057652    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls podName:612b8f12-28aa-444e-beb3-5700ebd338f3 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:32.057638254 +0000 UTC m=+39.011926301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls") pod "image-registry-76c4744ff8-jn9h4" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3") : secret "image-registry-tls" not found
Jul 10 18:40:30.748221 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:30.748182    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:30.828996 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:30.828959    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:31.989805 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:31.989709    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:32.049202 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:32.049163    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:32.075063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:32.075023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:32.075270 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:32.075113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:32.075270 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:32.075170    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:32.075270 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:32.075193    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c4744ff8-jn9h4: secret "image-registry-tls" not found
Jul 10 18:40:32.075270 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:32.075244    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:32.075270 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:32.075267    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls podName:612b8f12-28aa-444e-beb3-5700ebd338f3 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:36.075246058 +0000 UTC m=+43.029534105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls") pod "image-registry-76c4744ff8-jn9h4" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3") : secret "image-registry-tls" not found
Jul 10 18:40:32.075484 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:32.075286    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls podName:d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:36.075277811 +0000 UTC m=+43.029565856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls") pod "dns-default-k9d5q" (UID: "d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:33.853279 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:33.853197    2561 generic.go:356] "Generic (PLEG): container finished" podID="e48d716c-b5ee-4a31-b532-b815b3b58626" containerID="9b223e39b01abdad9e73e91867200aeccfc9e94219cdb71a5151042258914090" exitCode=0
Jul 10 18:40:33.853734 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:33.853283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerDied","Data":"9b223e39b01abdad9e73e91867200aeccfc9e94219cdb71a5151042258914090"}
Jul 10 18:40:33.854798 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:33.854774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rdwq2" event={"ID":"01dc8b49-c2dd-42af-a45d-9dac0564b514","Type":"ContainerStarted","Data":"44cbcb115267c328f17d31abd83000342dde297759917b94b6755761570d08e1"}
Jul 10 18:40:33.922962 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:33.922904    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rdwq2" podStartSLOduration=1.514949005 podStartE2EDuration="5.922885514s" totalImagesPullingTime="4.407936509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:33.92188816 +0000 UTC m=+40.876176272" watchObservedRunningTime="2026-07-10 18:40:33.922885514 +0000 UTC m=+40.877173559"
Jul 10 18:40:34.644845 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:34.644657    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:34.741804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:34.741772    2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log" limit=1024
Jul 10 18:40:34.862186 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:34.862136    2561 generic.go:356] "Generic (PLEG): container finished" podID="e48d716c-b5ee-4a31-b532-b815b3b58626" containerID="c920b2bbe7500e3b09e4a73b3f76bf6a4d705d319a054df959fbddc7d9eb822b" exitCode=0
Jul 10 18:40:34.862186 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:34.862180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerDied","Data":"c920b2bbe7500e3b09e4a73b3f76bf6a4d705d319a054df959fbddc7d9eb822b"}
Jul 10 18:40:35.867843 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:35.867801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" event={"ID":"e48d716c-b5ee-4a31-b532-b815b3b58626","Type":"ContainerStarted","Data":"7a61f893c6713a30c3010dbf20f8586368864d69ee89e0ab09c81ed6abec539a"}
Jul 10 18:40:35.894734 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:35.894678    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2gkfp" podStartSLOduration=8.345864571 podStartE2EDuration="41.894659337s" totalImagesPullingTime="33.548794766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:35.892451845 +0000 UTC m=+42.846739913" watchObservedRunningTime="2026-07-10 18:40:35.894659337 +0000 UTC m=+42.848947403"
Jul 10 18:40:36.109339 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:36.109298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:36.109508 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:36.109390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:36.109508 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:36.109462    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:36.109508 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:36.109478    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c4744ff8-jn9h4: secret "image-registry-tls" not found
Jul 10 18:40:36.109602 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:36.109530    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:36.109602 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:36.109536    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls podName:612b8f12-28aa-444e-beb3-5700ebd338f3 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:44.109519498 +0000 UTC m=+51.063807562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls") pod "image-registry-76c4744ff8-jn9h4" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3") : secret "image-registry-tls" not found
Jul 10 18:40:36.109602 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:36.109592    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls podName:d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:44.109579253 +0000 UTC m=+51.063867301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls") pod "dns-default-k9d5q" (UID: "d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:44.166310 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:44.166263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:40:44.166732 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:44.166334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:40:44.166732 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:44.166444    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:44.166732 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:44.166449    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:44.166732 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:44.166471    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c4744ff8-jn9h4: secret "image-registry-tls" not found
Jul 10 18:40:44.166732 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:44.166499    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls podName:d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.166485665 +0000 UTC m=+67.120773710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls") pod "dns-default-k9d5q" (UID: "d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:44.166732 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:44.166531    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls podName:612b8f12-28aa-444e-beb3-5700ebd338f3 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.166515573 +0000 UTC m=+67.120803618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls") pod "image-registry-76c4744ff8-jn9h4" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3") : secret "image-registry-tls" not found
Jul 10 18:40:51.385359 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.385321    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"]
Jul 10 18:40:51.387638 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.387618    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.390651 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.390628    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:51.391962 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.391935    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 10 18:40:51.392068 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.391974    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:51.392068 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.391974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-srqk8\""
Jul 10 18:40:51.392176 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.391948    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 10 18:40:51.396633 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.396614    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"]
Jul 10 18:40:51.415118 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.415091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac31c4b7-c733-47fd-8c91-8db7c161c319-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.415261 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.415127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac31c4b7-c733-47fd-8c91-8db7c161c319-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.415261 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.415223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt8mv\" (UniqueName: \"kubernetes.io/projected/ac31c4b7-c733-47fd-8c91-8db7c161c319-kube-api-access-vt8mv\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.498075 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.498039    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-4dz8j"]
Jul 10 18:40:51.499805 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.499788    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.503027 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.503004    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 10 18:40:51.503191 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.503125    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:40:51.503630 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.503606    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 10 18:40:51.503752 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.503612    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z7qc2\""
Jul 10 18:40:51.504514 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.504496    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:40:51.511586 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.511564    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 10 18:40:51.516128 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac31c4b7-c733-47fd-8c91-8db7c161c319-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.516204 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516155    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b43b9-931d-417d-bc0b-9609ead67393-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.516288 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgk59\" (UniqueName: \"kubernetes.io/projected/d91b43b9-931d-417d-bc0b-9609ead67393-kube-api-access-pgk59\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.516334 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac31c4b7-c733-47fd-8c91-8db7c161c319-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.516388 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b43b9-931d-417d-bc0b-9609ead67393-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.516430 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt8mv\" (UniqueName: \"kubernetes.io/projected/ac31c4b7-c733-47fd-8c91-8db7c161c319-kube-api-access-vt8mv\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.516520 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d91b43b9-931d-417d-bc0b-9609ead67393-snapshots\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.516573 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d91b43b9-931d-417d-bc0b-9609ead67393-serving-cert\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.516610 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d91b43b9-931d-417d-bc0b-9609ead67393-tmp\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.516733 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.516718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac31c4b7-c733-47fd-8c91-8db7c161c319-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.519327 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.519298    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-4dz8j"]
Jul 10 18:40:51.520682 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.520657    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac31c4b7-c733-47fd-8c91-8db7c161c319-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.530066 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.530035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt8mv\" (UniqueName: \"kubernetes.io/projected/ac31c4b7-c733-47fd-8c91-8db7c161c319-kube-api-access-vt8mv\") pod \"kube-storage-version-migrator-operator-799cb7f958-n8zbc\" (UID: \"ac31c4b7-c733-47fd-8c91-8db7c161c319\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.593588 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.593557    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"]
Jul 10 18:40:51.595389 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.595373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"
Jul 10 18:40:51.598545 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.598525    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:51.598798 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.598785    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:51.599039 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.599023    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-f4r88\""
Jul 10 18:40:51.616776 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.616741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"]
Jul 10 18:40:51.616965 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.616946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d91b43b9-931d-417d-bc0b-9609ead67393-snapshots\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.617052 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.616980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d91b43b9-931d-417d-bc0b-9609ead67393-serving-cert\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.617052 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.617020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d91b43b9-931d-417d-bc0b-9609ead67393-tmp\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.617229 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.617203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b43b9-931d-417d-bc0b-9609ead67393-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.617367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.617259    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgk59\" (UniqueName: \"kubernetes.io/projected/d91b43b9-931d-417d-bc0b-9609ead67393-kube-api-access-pgk59\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.617367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.617315    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b43b9-931d-417d-bc0b-9609ead67393-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.617367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.617345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr6d\" (UniqueName: \"kubernetes.io/projected/02fcf8cf-9311-4b87-a8ba-0d29f7a6f147-kube-api-access-dtr6d\") pod \"volume-data-source-validator-78495597f6-dn5rj\" (UID: \"02fcf8cf-9311-4b87-a8ba-0d29f7a6f147\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"
Jul 10 18:40:51.617524 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.617395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d91b43b9-931d-417d-bc0b-9609ead67393-tmp\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.617782 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.617758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d91b43b9-931d-417d-bc0b-9609ead67393-snapshots\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.618060 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.618039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b43b9-931d-417d-bc0b-9609ead67393-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.618139 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.618123    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d91b43b9-931d-417d-bc0b-9609ead67393-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.619824 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.619796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d91b43b9-931d-417d-bc0b-9609ead67393-serving-cert\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.620232 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.620214    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-jc755"]
Jul 10 18:40:51.622324 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.622305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.625957 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.625930    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 10 18:40:51.626070 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.626020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:51.626070 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.626033    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p8h5f\""
Jul 10 18:40:51.626285 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.626270    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 10 18:40:51.626530 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.626517    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 10 18:40:51.635012 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.634983    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 10 18:40:51.635880 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.635822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgk59\" (UniqueName: \"kubernetes.io/projected/d91b43b9-931d-417d-bc0b-9609ead67393-kube-api-access-pgk59\") pod \"insights-operator-68d9dd94cf-4dz8j\" (UID: \"d91b43b9-931d-417d-bc0b-9609ead67393\") " pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.641934 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.641907    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-jc755"]
Jul 10 18:40:51.696904 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.696862    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"
Jul 10 18:40:51.717835 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.717789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06ad978-e5e9-44ee-9250-b743cdeefd5e-config\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.717991 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.717842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/a06ad978-e5e9-44ee-9250-b743cdeefd5e-kube-api-access-4jznj\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.717991 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.717880    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr6d\" (UniqueName: \"kubernetes.io/projected/02fcf8cf-9311-4b87-a8ba-0d29f7a6f147-kube-api-access-dtr6d\") pod \"volume-data-source-validator-78495597f6-dn5rj\" (UID: \"02fcf8cf-9311-4b87-a8ba-0d29f7a6f147\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"
Jul 10 18:40:51.718087 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.717999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06ad978-e5e9-44ee-9250-b743cdeefd5e-serving-cert\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.718156 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.718123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a06ad978-e5e9-44ee-9250-b743cdeefd5e-trusted-ca\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.727753 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.727717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr6d\" (UniqueName: \"kubernetes.io/projected/02fcf8cf-9311-4b87-a8ba-0d29f7a6f147-kube-api-access-dtr6d\") pod \"volume-data-source-validator-78495597f6-dn5rj\" (UID: \"02fcf8cf-9311-4b87-a8ba-0d29f7a6f147\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"
Jul 10 18:40:51.809637 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.809606    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j"
Jul 10 18:40:51.819236 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.819199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06ad978-e5e9-44ee-9250-b743cdeefd5e-serving-cert\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.819386 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.819368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a06ad978-e5e9-44ee-9250-b743cdeefd5e-trusted-ca\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.819457 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.819406    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06ad978-e5e9-44ee-9250-b743cdeefd5e-config\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.819457 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.819444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/a06ad978-e5e9-44ee-9250-b743cdeefd5e-kube-api-access-4jznj\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.820462 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.820433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a06ad978-e5e9-44ee-9250-b743cdeefd5e-config\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.820557 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.820517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a06ad978-e5e9-44ee-9250-b743cdeefd5e-trusted-ca\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.822603 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.822577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a06ad978-e5e9-44ee-9250-b743cdeefd5e-serving-cert\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.823558 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.823529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc"]
Jul 10 18:40:51.827982 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.827960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jznj\" (UniqueName: \"kubernetes.io/projected/a06ad978-e5e9-44ee-9250-b743cdeefd5e-kube-api-access-4jznj\") pod \"console-operator-7866497494-jc755\" (UID: \"a06ad978-e5e9-44ee-9250-b743cdeefd5e\") " pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.828644 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:51.828605    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac31c4b7_c733_47fd_8c91_8db7c161c319.slice/crio-b2b825b0fa81c04614b4d00ffdb35a8e4bc41ccb615d1731de2bd64605dcf0a7 WatchSource:0}: Error finding container b2b825b0fa81c04614b4d00ffdb35a8e4bc41ccb615d1731de2bd64605dcf0a7: Status 404 returned error can't find the container with id b2b825b0fa81c04614b4d00ffdb35a8e4bc41ccb615d1731de2bd64605dcf0a7
Jul 10 18:40:51.904641 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.904602    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"
Jul 10 18:40:51.909155 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.909115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc" event={"ID":"ac31c4b7-c733-47fd-8c91-8db7c161c319","Type":"ContainerStarted","Data":"b2b825b0fa81c04614b4d00ffdb35a8e4bc41ccb615d1731de2bd64605dcf0a7"}
Jul 10 18:40:51.934540 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.934500    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:40:51.944295 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:51.944252    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-4dz8j"]
Jul 10 18:40:51.950232 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:51.950206    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91b43b9_931d_417d_bc0b_9609ead67393.slice/crio-b86f0cfd5eb7b1962ab33bb5682cc82b9dff29efb198bd0b56d3fffd12d9352d WatchSource:0}: Error finding container b86f0cfd5eb7b1962ab33bb5682cc82b9dff29efb198bd0b56d3fffd12d9352d: Status 404 returned error can't find the container with id b86f0cfd5eb7b1962ab33bb5682cc82b9dff29efb198bd0b56d3fffd12d9352d
Jul 10 18:40:52.049477 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:52.049435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj"]
Jul 10 18:40:52.054000 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:52.053937    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02fcf8cf_9311_4b87_a8ba_0d29f7a6f147.slice/crio-da6a3873235516fa449fd29f7809c3874f9d2c03a541d0de7b94a26ded2706ad WatchSource:0}: Error finding container da6a3873235516fa449fd29f7809c3874f9d2c03a541d0de7b94a26ded2706ad: Status 404 returned error can't find the container with id da6a3873235516fa449fd29f7809c3874f9d2c03a541d0de7b94a26ded2706ad
Jul 10 18:40:52.068478 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:52.068448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-jc755"]
Jul 10 18:40:52.072301 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:52.072273    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06ad978_e5e9_44ee_9250_b743cdeefd5e.slice/crio-09e2c4dfe0a288cc1c6926dcd8075afb7bd7c40a9fd3707b9e58e4c9c4205d9a WatchSource:0}: Error finding container 09e2c4dfe0a288cc1c6926dcd8075afb7bd7c40a9fd3707b9e58e4c9c4205d9a: Status 404 returned error can't find the container with id 09e2c4dfe0a288cc1c6926dcd8075afb7bd7c40a9fd3707b9e58e4c9c4205d9a
Jul 10 18:40:52.913581 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:52.913540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jc755" event={"ID":"a06ad978-e5e9-44ee-9250-b743cdeefd5e","Type":"ContainerStarted","Data":"09e2c4dfe0a288cc1c6926dcd8075afb7bd7c40a9fd3707b9e58e4c9c4205d9a"}
Jul 10 18:40:52.914868 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:52.914833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj" event={"ID":"02fcf8cf-9311-4b87-a8ba-0d29f7a6f147","Type":"ContainerStarted","Data":"da6a3873235516fa449fd29f7809c3874f9d2c03a541d0de7b94a26ded2706ad"}
Jul 10 18:40:52.916010 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:52.915982    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j" event={"ID":"d91b43b9-931d-417d-bc0b-9609ead67393","Type":"ContainerStarted","Data":"b86f0cfd5eb7b1962ab33bb5682cc82b9dff29efb198bd0b56d3fffd12d9352d"}
Jul 10 18:40:53.828875 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:53.828824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nltht"
Jul 10 18:40:53.920399 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:53.920363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj" event={"ID":"02fcf8cf-9311-4b87-a8ba-0d29f7a6f147","Type":"ContainerStarted","Data":"a213082394841de0e162fef072a357e2f1d26236e420fa4ed243fc552084d980"}
Jul 10 18:40:53.938346 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:53.938287    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-dn5rj" podStartSLOduration=1.660506044 podStartE2EDuration="2.938268797s" totalImagesPullingTime="1.277762753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:53.93747903 +0000 UTC m=+60.891767097" watchObservedRunningTime="2026-07-10 18:40:53.938268797 +0000 UTC m=+60.892556841"
Jul 10 18:40:55.927815 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:55.927774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j" event={"ID":"d91b43b9-931d-417d-bc0b-9609ead67393","Type":"ContainerStarted","Data":"2454f36e3b9b82d6bc930b9d76c73a842c1946da0d8b5759082122b12f64390f"}
Jul 10 18:40:55.929436 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:55.929404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc" event={"ID":"ac31c4b7-c733-47fd-8c91-8db7c161c319","Type":"ContainerStarted","Data":"f8ef6dd069f409805c8a85819e57114512d51a729e4f54d36eaa65be949f913a"}
Jul 10 18:40:55.931205 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:55.931182    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/0.log"
Jul 10 18:40:55.931304 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:55.931225    2561 generic.go:356] "Generic (PLEG): container finished" podID="a06ad978-e5e9-44ee-9250-b743cdeefd5e" containerID="46c64285a465aa41e0555c2ef52d5adff1f2c809dda36797e6a193ee9b3defc8" exitCode=255
Jul 10 18:40:55.931304 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:55.931267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jc755" event={"ID":"a06ad978-e5e9-44ee-9250-b743cdeefd5e","Type":"ContainerDied","Data":"46c64285a465aa41e0555c2ef52d5adff1f2c809dda36797e6a193ee9b3defc8"}
Jul 10 18:40:55.935445 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:55.931871    2561 scope.go:117] "RemoveContainer" containerID="46c64285a465aa41e0555c2ef52d5adff1f2c809dda36797e6a193ee9b3defc8"
Jul 10 18:40:55.957194 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:55.957112    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j" podStartSLOduration=1.713915536 podStartE2EDuration="4.957090594s" totalImagesPullingTime="3.243175058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:55.954472995 +0000 UTC m=+62.908761064" watchObservedRunningTime="2026-07-10 18:40:55.957090594 +0000 UTC m=+62.911378658"
Jul 10 18:40:56.844307 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.844239    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc" podStartSLOduration=2.487252316 podStartE2EDuration="5.844218201s" totalImagesPullingTime="3.356965885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:56.019535632 +0000 UTC m=+62.973823701" watchObservedRunningTime="2026-07-10 18:40:56.844218201 +0000 UTC m=+63.798506270"
Jul 10 18:40:56.844921 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.844905    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"]
Jul 10 18:40:56.869084 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.869049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"]
Jul 10 18:40:56.869249 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.869162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"
Jul 10 18:40:56.873632 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.873604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 10 18:40:56.873632 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.873604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 10 18:40:56.875065 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.875042    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4tjvz\""
Jul 10 18:40:56.936438 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.936410    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:40:56.936835 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.936821    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/0.log"
Jul 10 18:40:56.936872 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.936854    2561 generic.go:356] "Generic (PLEG): container finished" podID="a06ad978-e5e9-44ee-9250-b743cdeefd5e" containerID="cdee36b5cf536f4b3e4aaea5a8dcf0a9e5c133b44514e601e6aa305290248e92" exitCode=255
Jul 10 18:40:56.936974 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.936951    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jc755" event={"ID":"a06ad978-e5e9-44ee-9250-b743cdeefd5e","Type":"ContainerDied","Data":"cdee36b5cf536f4b3e4aaea5a8dcf0a9e5c133b44514e601e6aa305290248e92"}
Jul 10 18:40:56.937019 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.937010    2561 scope.go:117] "RemoveContainer" containerID="46c64285a465aa41e0555c2ef52d5adff1f2c809dda36797e6a193ee9b3defc8"
Jul 10 18:40:56.937373 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.937296    2561 scope.go:117] "RemoveContainer" containerID="cdee36b5cf536f4b3e4aaea5a8dcf0a9e5c133b44514e601e6aa305290248e92"
Jul 10 18:40:56.937491 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:56.937463    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jc755_openshift-console-operator(a06ad978-e5e9-44ee-9250-b743cdeefd5e)\"" pod="openshift-console-operator/console-operator-7866497494-jc755" podUID="a06ad978-e5e9-44ee-9250-b743cdeefd5e"
Jul 10 18:40:56.963834 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:56.963803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgbf\" (UniqueName: \"kubernetes.io/projected/5a4a213a-999c-402e-b83f-440a927acb98-kube-api-access-4hgbf\") pod \"migrator-7f59b54f6c-n6662\" (UID: \"5a4a213a-999c-402e-b83f-440a927acb98\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"
Jul 10 18:40:57.065657 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.065620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgbf\" (UniqueName: \"kubernetes.io/projected/5a4a213a-999c-402e-b83f-440a927acb98-kube-api-access-4hgbf\") pod \"migrator-7f59b54f6c-n6662\" (UID: \"5a4a213a-999c-402e-b83f-440a927acb98\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"
Jul 10 18:40:57.092914 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.092882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgbf\" (UniqueName: \"kubernetes.io/projected/5a4a213a-999c-402e-b83f-440a927acb98-kube-api-access-4hgbf\") pod \"migrator-7f59b54f6c-n6662\" (UID: \"5a4a213a-999c-402e-b83f-440a927acb98\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"
Jul 10 18:40:57.098973 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.098895    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t4x7p"]
Jul 10 18:40:57.102094 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.102066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.105245 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.105222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2ftqp\""
Jul 10 18:40:57.105505 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.105477    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:40:57.105667 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.105645    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:40:57.115522 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.115493    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t4x7p"]
Jul 10 18:40:57.166764 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.166730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-data-volume\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.166979 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.166831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-crio-socket\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.166979 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.166913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.166979 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.166968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.167084 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.166995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsbb\" (UniqueName: \"kubernetes.io/projected/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-api-access-7zsbb\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.179756 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.179722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"
Jul 10 18:40:57.267755 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.267714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.267927 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.267813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.267927 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.267835    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:40:57.267927 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.267846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsbb\" (UniqueName: \"kubernetes.io/projected/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-api-access-7zsbb\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.267927 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.267874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-data-volume\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.267927 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.267915    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:57.767891951 +0000 UTC m=+64.722180013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : configmap "kube-rbac-proxy" not found
Jul 10 18:40:57.268198 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.267972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-crio-socket\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.268198 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.267977    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:57.268198 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.268037    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:57.768021177 +0000 UTC m=+64.722309223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:57.268198 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.268072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-crio-socket\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.268365 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.268232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-data-volume\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.277980 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.277951    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsbb\" (UniqueName: \"kubernetes.io/projected/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-api-access-7zsbb\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.313664 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.313632    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662"]
Jul 10 18:40:57.316677 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:57.316649    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4a213a_999c_402e_b83f_440a927acb98.slice/crio-72222936244ffb23a9102b4a2efe321579a0cbb2757a81c9a5a6abef688cedf6 WatchSource:0}: Error finding container 72222936244ffb23a9102b4a2efe321579a0cbb2757a81c9a5a6abef688cedf6: Status 404 returned error can't find the container with id 72222936244ffb23a9102b4a2efe321579a0cbb2757a81c9a5a6abef688cedf6
Jul 10 18:40:57.772125 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.772073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.772299 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.772201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:57.772299 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.772260    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:57.772299 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.772285    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:40:57.772423 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.772330    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:58.772313252 +0000 UTC m=+65.726601298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:57.772423 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.772348    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:58.772340146 +0000 UTC m=+65.726628191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : configmap "kube-rbac-proxy" not found
Jul 10 18:40:57.940813 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.940777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662" event={"ID":"5a4a213a-999c-402e-b83f-440a927acb98","Type":"ContainerStarted","Data":"72222936244ffb23a9102b4a2efe321579a0cbb2757a81c9a5a6abef688cedf6"}
Jul 10 18:40:57.942282 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.942267    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:40:57.942619 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:57.942605    2561 scope.go:117] "RemoveContainer" containerID="cdee36b5cf536f4b3e4aaea5a8dcf0a9e5c133b44514e601e6aa305290248e92"
Jul 10 18:40:57.942757 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:57.942741    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jc755_openshift-console-operator(a06ad978-e5e9-44ee-9250-b743cdeefd5e)\"" pod="openshift-console-operator/console-operator-7866497494-jc755" podUID="a06ad978-e5e9-44ee-9250-b743cdeefd5e"
Jul 10 18:40:58.645732 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.645693    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-w2j6r"]
Jul 10 18:40:58.648100 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.648078    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.650839 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.650814    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 10 18:40:58.651025 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.651000    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 10 18:40:58.651130 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.651071    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 10 18:40:58.652129 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.652107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ljgjm\""
Jul 10 18:40:58.652241 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.652225    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 10 18:40:58.659171 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.659112    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-w2j6r"]
Jul 10 18:40:58.681248 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.681164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d84e510-1b64-408a-84c7-7cf7512965c5-signing-key\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.681394 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.681255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lr6g\" (UniqueName: \"kubernetes.io/projected/1d84e510-1b64-408a-84c7-7cf7512965c5-kube-api-access-6lr6g\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.681394 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.681292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d84e510-1b64-408a-84c7-7cf7512965c5-signing-cabundle\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.782213 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.782180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lr6g\" (UniqueName: \"kubernetes.io/projected/1d84e510-1b64-408a-84c7-7cf7512965c5-kube-api-access-6lr6g\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.782341 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.782263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d84e510-1b64-408a-84c7-7cf7512965c5-signing-cabundle\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.782341 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.782306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:58.782459 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.782404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:40:58.782459 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.782451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d84e510-1b64-408a-84c7-7cf7512965c5-signing-key\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.782543 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:58.782475    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:40:58.782543 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:58.782494    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:58.782644 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:58.782548    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.782527976 +0000 UTC m=+67.736816036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : configmap "kube-rbac-proxy" not found
Jul 10 18:40:58.782644 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:58.782569    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.782558597 +0000 UTC m=+67.736846644 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:58.783055 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.783033    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d84e510-1b64-408a-84c7-7cf7512965c5-signing-cabundle\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.784868 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.784844    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d84e510-1b64-408a-84c7-7cf7512965c5-signing-key\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.794711 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.794682    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lr6g\" (UniqueName: \"kubernetes.io/projected/1d84e510-1b64-408a-84c7-7cf7512965c5-kube-api-access-6lr6g\") pod \"service-ca-79f68b9f4f-w2j6r\" (UID: \"1d84e510-1b64-408a-84c7-7cf7512965c5\") " pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.948253 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.948167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662" event={"ID":"5a4a213a-999c-402e-b83f-440a927acb98","Type":"ContainerStarted","Data":"60cfe6811c9510dcfb7c7531271923d4000a28845cdadcae0dd7a80037026d4c"}
Jul 10 18:40:58.948253 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.948220    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662" event={"ID":"5a4a213a-999c-402e-b83f-440a927acb98","Type":"ContainerStarted","Data":"3b72528bf62c2fcefe3365b9522ca66daf61839a9f84e58adadd67667f15d4f8"}
Jul 10 18:40:58.959897 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.959873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r"
Jul 10 18:40:58.968197 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:58.968104    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-n6662" podStartSLOduration=1.585546253 podStartE2EDuration="2.968087366s" totalImagesPullingTime="1.382541113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:58.967138447 +0000 UTC m=+65.921426515" watchObservedRunningTime="2026-07-10 18:40:58.968087366 +0000 UTC m=+65.922375432"
Jul 10 18:40:59.081133 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.081104    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-w2j6r"]
Jul 10 18:40:59.084033 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:59.084008    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d84e510_1b64_408a_84c7_7cf7512965c5.slice/crio-918ae50b444e3dd04f28f67174fc036996d60557aa90af832a172679e6d47805 WatchSource:0}: Error finding container 918ae50b444e3dd04f28f67174fc036996d60557aa90af832a172679e6d47805: Status 404 returned error can't find the container with id 918ae50b444e3dd04f28f67174fc036996d60557aa90af832a172679e6d47805
Jul 10 18:40:59.287476 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.287376    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:59.290564 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.290543    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:40:59.300560 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.300535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3c91b6f5-d69d-47d5-9782-18d0a40d6f4e-original-pull-secret\") pod \"global-pull-secret-syncer-krjlw\" (UID: \"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e\") " pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:59.388565 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.388527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:40:59.391597 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.391577    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:40:59.399042 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:59.399021    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 10 18:40:59.399140 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:40:59.399088    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs podName:f6e8e427-7654-41f4-a0cf-8b4e07fc6381 nodeName:}" failed. No retries permitted until 2026-07-10 18:42:03.39906704 +0000 UTC m=+130.353355085 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs") pod "network-metrics-daemon-4sbhh" (UID: "f6e8e427-7654-41f4-a0cf-8b4e07fc6381") : secret "metrics-daemon-secret" not found
Jul 10 18:40:59.456319 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.456276    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krjlw"
Jul 10 18:40:59.489907 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.489871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:59.493725 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.493696    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:40:59.504500 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.504473    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:40:59.514690 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.514660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmdt\" (UniqueName: \"kubernetes.io/projected/6b52971f-2c3f-4986-a372-2c8d98ea58da-kube-api-access-fnmdt\") pod \"network-check-target-fjdnw\" (UID: \"6b52971f-2c3f-4986-a372-2c8d98ea58da\") " pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:59.616102 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.615988    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krjlw"]
Jul 10 18:40:59.619219 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:59.619192    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c91b6f5_d69d_47d5_9782_18d0a40d6f4e.slice/crio-109d7048a107c0bdedca6d54ee695aff201b1bacdc5c8bf9c653cee290aa111e WatchSource:0}: Error finding container 109d7048a107c0bdedca6d54ee695aff201b1bacdc5c8bf9c653cee290aa111e: Status 404 returned error can't find the container with id 109d7048a107c0bdedca6d54ee695aff201b1bacdc5c8bf9c653cee290aa111e
Jul 10 18:40:59.703930 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.703900    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hhb4q_ee32311a-4aa2-4168-bffd-f355e8dae244/dns-node-resolver/0.log"
Jul 10 18:40:59.764348 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.764318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-p7x9n\""
Jul 10 18:40:59.772223 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.772194    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:40:59.928266 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.928236    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjdnw"]
Jul 10 18:40:59.931829 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:40:59.931802    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b52971f_2c3f_4986_a372_2c8d98ea58da.slice/crio-c2f5089737fbbb2c37858efa4930b6055e24549701db0bc97f48ef39b76a81b6 WatchSource:0}: Error finding container c2f5089737fbbb2c37858efa4930b6055e24549701db0bc97f48ef39b76a81b6: Status 404 returned error can't find the container with id c2f5089737fbbb2c37858efa4930b6055e24549701db0bc97f48ef39b76a81b6
Jul 10 18:40:59.953074 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.953036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r" event={"ID":"1d84e510-1b64-408a-84c7-7cf7512965c5","Type":"ContainerStarted","Data":"918ae50b444e3dd04f28f67174fc036996d60557aa90af832a172679e6d47805"}
Jul 10 18:40:59.954486 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.954459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjdnw" event={"ID":"6b52971f-2c3f-4986-a372-2c8d98ea58da","Type":"ContainerStarted","Data":"c2f5089737fbbb2c37858efa4930b6055e24549701db0bc97f48ef39b76a81b6"}
Jul 10 18:40:59.955752 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:40:59.955724    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krjlw" event={"ID":"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e","Type":"ContainerStarted","Data":"109d7048a107c0bdedca6d54ee695aff201b1bacdc5c8bf9c653cee290aa111e"}
Jul 10 18:41:00.198100 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:00.198063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:41:00.198303 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:00.198235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:41:00.198303 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.198255    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:41:00.198432 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.198332    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls podName:d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:32.198311809 +0000 UTC m=+99.152599871 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls") pod "dns-default-k9d5q" (UID: "d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1") : secret "dns-default-metrics-tls" not found
Jul 10 18:41:00.198432 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.198380    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:41:00.198432 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.198395    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c4744ff8-jn9h4: secret "image-registry-tls" not found
Jul 10 18:41:00.198569 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.198443    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls podName:612b8f12-28aa-444e-beb3-5700ebd338f3 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:32.198428309 +0000 UTC m=+99.152716354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls") pod "image-registry-76c4744ff8-jn9h4" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3") : secret "image-registry-tls" not found
Jul 10 18:41:00.302965 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:00.302935    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8cvcp_3f921aa9-6e25-4f9c-a45c-3eb718f655a9/node-ca/0.log"
Jul 10 18:41:00.802757 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:00.802719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:00.802937 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:00.802822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:00.802937 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.802864    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:00.803030 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.802944    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:04.802923765 +0000 UTC m=+71.757211824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:00.803030 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.802952    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.803030 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:00.802990    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:04.802980179 +0000 UTC m=+71.757268230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.966177 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:00.965020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r" event={"ID":"1d84e510-1b64-408a-84c7-7cf7512965c5","Type":"ContainerStarted","Data":"f86bed1c215c7e9f1306e5378aa3604ab85d0c0ca35331451da6e0976cff8029"}
Jul 10 18:41:00.985982 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:00.985915    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-w2j6r" podStartSLOduration=1.260578166 podStartE2EDuration="2.985893333s" totalImagesPullingTime="1.725315167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:00.984974732 +0000 UTC m=+67.939262821" watchObservedRunningTime="2026-07-10 18:41:00.985893333 +0000 UTC m=+67.940181403"
Jul 10 18:41:01.101479 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:01.101389    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rdwq2_01dc8b49-c2dd-42af-a45d-9dac0564b514/serve-healthcheck-canary/0.log"
Jul 10 18:41:01.512578 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:01.512485    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-n6662_5a4a213a-999c-402e-b83f-440a927acb98/migrator/0.log"
Jul 10 18:41:01.702888 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:01.702856    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-n6662_5a4a213a-999c-402e-b83f-440a927acb98/graceful-termination/0.log"
Jul 10 18:41:01.907033 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:01.906959    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-n8zbc_ac31c4b7-c733-47fd-8c91-8db7c161c319/kube-storage-version-migrator-operator/0.log"
Jul 10 18:41:01.935134 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:01.935096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:41:01.935330 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:01.935183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:41:01.935650 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:01.935633    2561 scope.go:117] "RemoveContainer" containerID="cdee36b5cf536f4b3e4aaea5a8dcf0a9e5c133b44514e601e6aa305290248e92"
Jul 10 18:41:01.936272 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:01.935829    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-jc755_openshift-console-operator(a06ad978-e5e9-44ee-9250-b743cdeefd5e)\"" pod="openshift-console-operator/console-operator-7866497494-jc755" podUID="a06ad978-e5e9-44ee-9250-b743cdeefd5e"
Jul 10 18:41:04.841217 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:04.841177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:04.841662 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:04.841286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:04.841662 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:04.841377    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:04.841662 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:04.841393    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:04.841662 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:04.841453    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:12.841435373 +0000 UTC m=+79.795723419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:04.841662 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:04.841477    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:12.841465718 +0000 UTC m=+79.795753796 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:05.989358 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:05.989326    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjdnw" event={"ID":"6b52971f-2c3f-4986-a372-2c8d98ea58da","Type":"ContainerStarted","Data":"e73b0db7e6f31ebf5e75fe690cdd97aa3a03d8bac63259b352185bf9eda0b2ed"}
Jul 10 18:41:05.989809 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:05.989418    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:41:05.990896 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:05.990861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krjlw" event={"ID":"3c91b6f5-d69d-47d5-9782-18d0a40d6f4e","Type":"ContainerStarted","Data":"370c444b5606458c0eab109f084d7a5673cd1c1bdfae4ac67594eed938cc1801"}
Jul 10 18:41:06.012012 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:06.011958    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fjdnw" podStartSLOduration=66.583163801 podStartE2EDuration="1m12.011944436s" totalImagesPullingTime="5.428780635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:06.011693885 +0000 UTC m=+72.965981953" watchObservedRunningTime="2026-07-10 18:41:06.011944436 +0000 UTC m=+72.966232502"
Jul 10 18:41:06.031951 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:06.031896    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-krjlw" podStartSLOduration=66.289164019 podStartE2EDuration="1m12.031879039s" totalImagesPullingTime="5.74271502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:06.030732801 +0000 UTC m=+72.985020871" watchObservedRunningTime="2026-07-10 18:41:06.031879039 +0000 UTC m=+72.986167106"
Jul 10 18:41:12.908466 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:12.908417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:12.908947 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:12.908555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:12.908947 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:12.908697    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:12.908947 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:12.908758    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls podName:29dabca5-17b1-40dd-9a4b-f8f473ef8d52 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:28.908742673 +0000 UTC m=+95.863030722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t4x7p" (UID: "29dabca5-17b1-40dd-9a4b-f8f473ef8d52") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:12.909117 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:12.909031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:16.535810 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:16.535695    2561 scope.go:117] "RemoveContainer" containerID="cdee36b5cf536f4b3e4aaea5a8dcf0a9e5c133b44514e601e6aa305290248e92"
Jul 10 18:41:17.027507 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:17.027477    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:41:17.027678 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:17.027534    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-jc755" event={"ID":"a06ad978-e5e9-44ee-9250-b743cdeefd5e","Type":"ContainerStarted","Data":"9653ad1d462fa6a76b349924fa11f0c8cc997a37504848a7ef8221a8c14d6362"}
Jul 10 18:41:17.027852 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:17.027830    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:41:17.048536 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:17.048472    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-jc755" podStartSLOduration=22.926022023 podStartE2EDuration="26.048452065s" totalImagesPullingTime="3.122430042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:17.047814889 +0000 UTC m=+84.002102956" watchObservedRunningTime="2026-07-10 18:41:17.048452065 +0000 UTC m=+84.002740132"
Jul 10 18:41:17.130542 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:17.130509    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-jc755"
Jul 10 18:41:20.831300 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.831267    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"]
Jul 10 18:41:20.834654 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.834628    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:20.838248 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.838224    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 10 18:41:20.838409 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.838225    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 18:41:20.838409 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.838298    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 18:41:20.838409 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.838226    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 18:41:20.838409 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.838225    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 10 18:41:20.839379 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.839360    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 10 18:41:20.839497 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.839438    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 10 18:41:20.855868 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.855838    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"]
Jul 10 18:41:20.971367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.971326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:20.971367 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.971366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19aa79-6467-45cf-b67e-a717adc4d486-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:20.971593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.971460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bspd\" (UniqueName: \"kubernetes.io/projected/3e19aa79-6467-45cf-b67e-a717adc4d486-kube-api-access-6bspd\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:20.971593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.971501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-hub\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:20.971593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.971524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:20.971593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:20.971578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-ca\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.072455 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.072407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.072455 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.072458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19aa79-6467-45cf-b67e-a717adc4d486-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.072721 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.072518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bspd\" (UniqueName: \"kubernetes.io/projected/3e19aa79-6467-45cf-b67e-a717adc4d486-kube-api-access-6bspd\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.072721 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.072547    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-hub\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.072721 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.072576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.072721 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.072620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-ca\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.073399 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.073367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e19aa79-6467-45cf-b67e-a717adc4d486-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.075424 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.075402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.075529 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.075463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-hub\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.075529 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.075463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-ca\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.075529 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.075508    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e19aa79-6467-45cf-b67e-a717adc4d486-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.094516 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.094431    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bspd\" (UniqueName: \"kubernetes.io/projected/3e19aa79-6467-45cf-b67e-a717adc4d486-kube-api-access-6bspd\") pod \"cluster-proxy-proxy-agent-55d9b98fb8-hvwfz\" (UID: \"3e19aa79-6467-45cf-b67e-a717adc4d486\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.157407 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.157370    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"
Jul 10 18:41:21.332428 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:21.332394    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz"]
Jul 10 18:41:21.335524 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:41:21.335481    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e19aa79_6467_45cf_b67e_a717adc4d486.slice/crio-7295f8e63cdd3182079bc3f190b9a8413394f855e4c82ce5f44fd6aa69222ba1 WatchSource:0}: Error finding container 7295f8e63cdd3182079bc3f190b9a8413394f855e4c82ce5f44fd6aa69222ba1: Status 404 returned error can't find the container with id 7295f8e63cdd3182079bc3f190b9a8413394f855e4c82ce5f44fd6aa69222ba1
Jul 10 18:41:22.045523 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:22.045446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz" event={"ID":"3e19aa79-6467-45cf-b67e-a717adc4d486","Type":"ContainerStarted","Data":"7295f8e63cdd3182079bc3f190b9a8413394f855e4c82ce5f44fd6aa69222ba1"}
Jul 10 18:41:25.059053 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:25.059022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz" event={"ID":"3e19aa79-6467-45cf-b67e-a717adc4d486","Type":"ContainerStarted","Data":"98c4df92e3e75cb6ad0360b9c0506075e104506169b2470409d0360f9970ed89"}
Jul 10 18:41:25.059427 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:25.059068    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz" event={"ID":"3e19aa79-6467-45cf-b67e-a717adc4d486","Type":"ContainerStarted","Data":"675cbb8409dad72dc1a0f345301b12197f7f6db83bf1a5e85128291e9125c85d"}
Jul 10 18:41:26.064247 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:26.064212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz" event={"ID":"3e19aa79-6467-45cf-b67e-a717adc4d486","Type":"ContainerStarted","Data":"9a7a72dfc408807d03a5c114399f23936903deb105e624f3c9df27c95d3aff93"}
Jul 10 18:41:26.088823 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:26.088772    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55d9b98fb8-hvwfz" podStartSLOduration=2.512989033 podStartE2EDuration="6.088755155s" totalImagesPullingTime="3.575766122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:26.087419749 +0000 UTC m=+93.041707835" watchObservedRunningTime="2026-07-10 18:41:26.088755155 +0000 UTC m=+93.043043221"
Jul 10 18:41:28.947140 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:28.947077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:28.949428 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:28.949403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29dabca5-17b1-40dd-9a4b-f8f473ef8d52-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t4x7p\" (UID: \"29dabca5-17b1-40dd-9a4b-f8f473ef8d52\") " pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:29.211690 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:29.211601    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t4x7p"
Jul 10 18:41:29.347838 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:29.347795    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t4x7p"]
Jul 10 18:41:29.351886 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:41:29.351861    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dabca5_17b1_40dd_9a4b_f8f473ef8d52.slice/crio-04da4bc531c8ab58424e7643986ffe6122c4217ad31600c612ccfdd5ca16d40c WatchSource:0}: Error finding container 04da4bc531c8ab58424e7643986ffe6122c4217ad31600c612ccfdd5ca16d40c: Status 404 returned error can't find the container with id 04da4bc531c8ab58424e7643986ffe6122c4217ad31600c612ccfdd5ca16d40c
Jul 10 18:41:30.081111 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:30.081013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4x7p" event={"ID":"29dabca5-17b1-40dd-9a4b-f8f473ef8d52","Type":"ContainerStarted","Data":"04da4bc531c8ab58424e7643986ffe6122c4217ad31600c612ccfdd5ca16d40c"}
Jul 10 18:41:32.279879 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.279836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:41:32.280428 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.279966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:41:32.282631 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.282605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"image-registry-76c4744ff8-jn9h4\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") " pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:41:32.282760 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.282651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1-metrics-tls\") pod \"dns-default-k9d5q\" (UID: \"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1\") " pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:41:32.552614 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.552541    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-znrdn\""
Jul 10 18:41:32.560116 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.560082    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:41:32.569328 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.569271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-stn59\""
Jul 10 18:41:32.577113 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:32.577083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:41:33.235500 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:33.233296    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9d5q"]
Jul 10 18:41:33.252471 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:33.252435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c4744ff8-jn9h4"]
Jul 10 18:41:33.255535 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:41:33.255501    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612b8f12_28aa_444e_beb3_5700ebd338f3.slice/crio-ca5a1fc256b3d564c75571401092e8eccc6b5612aebb77d2187cf19e48ac86ff WatchSource:0}: Error finding container ca5a1fc256b3d564c75571401092e8eccc6b5612aebb77d2187cf19e48ac86ff: Status 404 returned error can't find the container with id ca5a1fc256b3d564c75571401092e8eccc6b5612aebb77d2187cf19e48ac86ff
Jul 10 18:41:34.095018 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:34.094971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9d5q" event={"ID":"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1","Type":"ContainerStarted","Data":"b6d5356bfbf6696e930a929645ace3192586f50c63d8577288c7a0bb975d922e"}
Jul 10 18:41:34.097348 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:34.097313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4x7p" event={"ID":"29dabca5-17b1-40dd-9a4b-f8f473ef8d52","Type":"ContainerStarted","Data":"e7ffe3086023c2a158b87f59fc8aa41583d60895779e3738bfbc21711d84d975"}
Jul 10 18:41:34.097502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:34.097382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4x7p" event={"ID":"29dabca5-17b1-40dd-9a4b-f8f473ef8d52","Type":"ContainerStarted","Data":"cd4a4247c2615eeab4d82a039f4842b5233d3dacd42161d7efc5ee518b1d859c"}
Jul 10 18:41:34.099213 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:34.099172    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4" event={"ID":"612b8f12-28aa-444e-beb3-5700ebd338f3","Type":"ContainerStarted","Data":"d505532f042f83f1728e8ce8f7bb5bd725c7f29622466906bd387a0c3c293ba9"}
Jul 10 18:41:34.099341 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:34.099225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4" event={"ID":"612b8f12-28aa-444e-beb3-5700ebd338f3","Type":"ContainerStarted","Data":"ca5a1fc256b3d564c75571401092e8eccc6b5612aebb77d2187cf19e48ac86ff"}
Jul 10 18:41:34.099341 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:34.099305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:41:34.127572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:34.127520    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4" podStartSLOduration=100.127503017 podStartE2EDuration="1m40.127503017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:34.126859815 +0000 UTC m=+101.081147901" watchObservedRunningTime="2026-07-10 18:41:34.127503017 +0000 UTC m=+101.081791083"
Jul 10 18:41:36.108965 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:36.108917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9d5q" event={"ID":"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1","Type":"ContainerStarted","Data":"510a4f8695aa362397825564d15e4000e025d8d9db3dce4bff3db3fd8a3af6ad"}
Jul 10 18:41:36.109413 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:36.109009    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:41:36.109413 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:36.109044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9d5q" event={"ID":"d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1","Type":"ContainerStarted","Data":"45d00bbc4bde5a49ff3edd6d4a01064a54d4cb3b87f6be984573592999ae43c9"}
Jul 10 18:41:36.131112 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:36.131056    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k9d5q" podStartSLOduration=66.218092894 podStartE2EDuration="1m8.131038676s" totalImagesPullingTime="1.912945782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:36.12997202 +0000 UTC m=+103.084260087" watchObservedRunningTime="2026-07-10 18:41:36.131038676 +0000 UTC m=+103.085326742"
Jul 10 18:41:36.997216 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:36.997183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fjdnw"
Jul 10 18:41:38.118115 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:38.118028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t4x7p" event={"ID":"29dabca5-17b1-40dd-9a4b-f8f473ef8d52","Type":"ContainerStarted","Data":"17dcdc39d1000c9e84fcf2283911e486378e009e7b54b19fca158ac06a5be5aa"}
Jul 10 18:41:38.138753 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:38.138678    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t4x7p" podStartSLOduration=32.943234243 podStartE2EDuration="41.138655902s" totalImagesPullingTime="8.195421659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:38.138616462 +0000 UTC m=+105.092904541" watchObservedRunningTime="2026-07-10 18:41:38.138655902 +0000 UTC m=+105.092943964"
Jul 10 18:41:41.628935 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.628898    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tmm6s"]
Jul 10 18:41:41.631643 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.631624    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.636465 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.636436    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 18:41:41.636465 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.636454    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 18:41:41.636672 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.636498    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 18:41:41.645254 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.645218    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 18:41:41.645436 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.645225    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 18:41:41.645436 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.645337    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 18:41:41.645436 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.645374    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-psdxh\""
Jul 10 18:41:41.646772 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.646752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j65mm\" (UniqueName: \"kubernetes.io/projected/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-kube-api-access-j65mm\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.646849 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.646793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.646849 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.646818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-textfile\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.646849 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.646833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-wtmp\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.646994 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.646854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.646994 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.646881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-tls\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.647093 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.646994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-sys\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.647093 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.647025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-root\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.647228 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.647094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-metrics-client-ca\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748166 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-textfile\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748373 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748177    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-wtmp\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748373 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748373 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748339    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-tls\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748542 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-wtmp\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-sys\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748593 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:41.748556    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 10 18:41:41.748593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-textfile\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-root\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748780 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:41:41.748633    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-tls podName:76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a nodeName:}" failed. No retries permitted until 2026-07-10 18:41:42.248612677 +0000 UTC m=+109.202900728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-tls") pod "node-exporter-tmm6s" (UID: "76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a") : secret "node-exporter-tls" not found
Jul 10 18:41:41.748780 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-root\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748780 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-metrics-client-ca\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748780 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j65mm\" (UniqueName: \"kubernetes.io/projected/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-kube-api-access-j65mm\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.748949 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.748797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.749198 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.749176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-sys\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.749407 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.749386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-metrics-client-ca\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.749459 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.749385    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.750982 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.750959    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:41.760704 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:41.760667    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j65mm\" (UniqueName: \"kubernetes.io/projected/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-kube-api-access-j65mm\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:42.252344 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:42.252305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-tls\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:42.254734 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:42.254698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a-node-exporter-tls\") pod \"node-exporter-tmm6s\" (UID: \"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a\") " pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:42.540838 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:42.540744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tmm6s"
Jul 10 18:41:42.553681 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:41:42.553638    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c6b8d8_f57b_4e2a_9a4d_ead51ed0435a.slice/crio-8bca4b6b18854f7146ea2ee3efdabc790843f83861e0415b886f25ed1cfc6dc5 WatchSource:0}: Error finding container 8bca4b6b18854f7146ea2ee3efdabc790843f83861e0415b886f25ed1cfc6dc5: Status 404 returned error can't find the container with id 8bca4b6b18854f7146ea2ee3efdabc790843f83861e0415b886f25ed1cfc6dc5
Jul 10 18:41:43.136010 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:43.135972    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmm6s" event={"ID":"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a","Type":"ContainerStarted","Data":"8bca4b6b18854f7146ea2ee3efdabc790843f83861e0415b886f25ed1cfc6dc5"}
Jul 10 18:41:43.666449 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:43.666395    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c4744ff8-jn9h4"]
Jul 10 18:41:44.141338 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:44.141300    2561 generic.go:356] "Generic (PLEG): container finished" podID="76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a" containerID="fb48ca001922dc0da468fac0ad3cef77d68a81a1eab8eaee2f57a1fd37f12b52" exitCode=0
Jul 10 18:41:44.141816 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:44.141372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmm6s" event={"ID":"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a","Type":"ContainerDied","Data":"fb48ca001922dc0da468fac0ad3cef77d68a81a1eab8eaee2f57a1fd37f12b52"}
Jul 10 18:41:45.146894 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:45.146847    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmm6s" event={"ID":"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a","Type":"ContainerStarted","Data":"486b814609943501120746da94cb1a1642f0a1f28a46cb6105dd84c7d44e7ce4"}
Jul 10 18:41:45.147321 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:45.146947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tmm6s" event={"ID":"76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a","Type":"ContainerStarted","Data":"cdbeec87825ef6d40d4a0e147cdfd5c70227cffe5fb5ee53ae39679a134086b6"}
Jul 10 18:41:45.172385 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:45.172332    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tmm6s" podStartSLOduration=3.154781587 podStartE2EDuration="4.17231486s" totalImagesPullingTime="947.808178ms" totalInitContainerRuntime="69.725095ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:45.170490778 +0000 UTC m=+112.124778847" watchObservedRunningTime="2026-07-10 18:41:45.17231486 +0000 UTC m=+112.126602927"
Jul 10 18:41:46.115131 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:46.115099    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k9d5q"
Jul 10 18:41:53.672201 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:41:53.672169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:42:03.431720 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:03.431676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:42:03.434097 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:03.434069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6e8e427-7654-41f4-a0cf-8b4e07fc6381-metrics-certs\") pod \"network-metrics-daemon-4sbhh\" (UID: \"f6e8e427-7654-41f4-a0cf-8b4e07fc6381\") " pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:42:03.648568 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:03.648535    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v9r2k\""
Jul 10 18:42:03.656002 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:03.655972    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sbhh"
Jul 10 18:42:03.800886 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:03.800857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sbhh"]
Jul 10 18:42:03.803535 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:42:03.803509    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e8e427_7654_41f4_a0cf_8b4e07fc6381.slice/crio-10d80104b5148fcfcc87fd39dc67e2760cfbecc2e1e7b492c11275b0e7555bbd WatchSource:0}: Error finding container 10d80104b5148fcfcc87fd39dc67e2760cfbecc2e1e7b492c11275b0e7555bbd: Status 404 returned error can't find the container with id 10d80104b5148fcfcc87fd39dc67e2760cfbecc2e1e7b492c11275b0e7555bbd
Jul 10 18:42:04.213047 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:04.213008    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sbhh" event={"ID":"f6e8e427-7654-41f4-a0cf-8b4e07fc6381","Type":"ContainerStarted","Data":"10d80104b5148fcfcc87fd39dc67e2760cfbecc2e1e7b492c11275b0e7555bbd"}
Jul 10 18:42:05.218980 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:05.218948    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sbhh" event={"ID":"f6e8e427-7654-41f4-a0cf-8b4e07fc6381","Type":"ContainerStarted","Data":"486014a9ed6f17eba188d3b9fb290841331431a6a813bcc2b35f33cf6b19b1f8"}
Jul 10 18:42:05.219372 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:05.219002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sbhh" event={"ID":"f6e8e427-7654-41f4-a0cf-8b4e07fc6381","Type":"ContainerStarted","Data":"7a422b6d8be1a95aef11b942d6054037061de524a36456c7b5f4e800c5c2e500"}
Jul 10 18:42:05.238505 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:05.238448    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4sbhh" podStartSLOduration=130.29427582 podStartE2EDuration="2m11.238429999s" totalImagesPullingTime="944.154179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:42:05.237244364 +0000 UTC m=+132.191532456" watchObservedRunningTime="2026-07-10 18:42:05.238429999 +0000 UTC m=+132.192718044"
Jul 10 18:42:08.686638 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:08.686583    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4" podUID="612b8f12-28aa-444e-beb3-5700ebd338f3" containerName="registry" containerID="cri-o://d505532f042f83f1728e8ce8f7bb5bd725c7f29622466906bd387a0c3c293ba9" gracePeriod=30
Jul 10 18:42:09.234850 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.234810    2561 generic.go:356] "Generic (PLEG): container finished" podID="612b8f12-28aa-444e-beb3-5700ebd338f3" containerID="d505532f042f83f1728e8ce8f7bb5bd725c7f29622466906bd387a0c3c293ba9" exitCode=0
Jul 10 18:42:09.235030 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.234899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4" event={"ID":"612b8f12-28aa-444e-beb3-5700ebd338f3","Type":"ContainerDied","Data":"d505532f042f83f1728e8ce8f7bb5bd725c7f29622466906bd387a0c3c293ba9"}
Jul 10 18:42:09.510717 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.510643    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:42:09.582289 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582480 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582305    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-certificates\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582480 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582352    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-installation-pull-secrets\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582480 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582384    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-bound-sa-token\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582480 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582423    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-image-registry-private-configuration\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582480 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582467    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/612b8f12-28aa-444e-beb3-5700ebd338f3-ca-trust-extracted\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582735 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582510    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8c92\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-kube-api-access-v8c92\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582735 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582540    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-trusted-ca\") pod \"612b8f12-28aa-444e-beb3-5700ebd338f3\" (UID: \"612b8f12-28aa-444e-beb3-5700ebd338f3\") "
Jul 10 18:42:09.582971 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.582899    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:09.583116 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.583091    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:42:09.584955 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.584928    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:09.585067 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.584967    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-kube-api-access-v8c92" (OuterVolumeSpecName: "kube-api-access-v8c92") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "kube-api-access-v8c92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:09.585067 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.585036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:09.585210 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.585125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:42:09.585263 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.585212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:42:09.591291 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.591261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612b8f12-28aa-444e-beb3-5700ebd338f3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "612b8f12-28aa-444e-beb3-5700ebd338f3" (UID: "612b8f12-28aa-444e-beb3-5700ebd338f3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:42:09.683874 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683833    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-certificates\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:09.683874 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683865    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-installation-pull-secrets\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:09.683874 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683875    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-bound-sa-token\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:09.684102 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683885    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/612b8f12-28aa-444e-beb3-5700ebd338f3-image-registry-private-configuration\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:09.684102 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683896    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/612b8f12-28aa-444e-beb3-5700ebd338f3-ca-trust-extracted\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:09.684102 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683905    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8c92\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-kube-api-access-v8c92\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:09.684102 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683913    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/612b8f12-28aa-444e-beb3-5700ebd338f3-trusted-ca\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:09.684102 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:09.683921    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/612b8f12-28aa-444e-beb3-5700ebd338f3-registry-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:42:10.239957 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:10.239924    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4"
Jul 10 18:42:10.240498 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:10.239920    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c4744ff8-jn9h4" event={"ID":"612b8f12-28aa-444e-beb3-5700ebd338f3","Type":"ContainerDied","Data":"ca5a1fc256b3d564c75571401092e8eccc6b5612aebb77d2187cf19e48ac86ff"}
Jul 10 18:42:10.240498 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:10.240053    2561 scope.go:117] "RemoveContainer" containerID="d505532f042f83f1728e8ce8f7bb5bd725c7f29622466906bd387a0c3c293ba9"
Jul 10 18:42:10.263508 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:10.263480    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c4744ff8-jn9h4"]
Jul 10 18:42:10.268034 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:10.268008    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76c4744ff8-jn9h4"]
Jul 10 18:42:11.246898 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:11.246864    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac31c4b7-c733-47fd-8c91-8db7c161c319" containerID="f8ef6dd069f409805c8a85819e57114512d51a729e4f54d36eaa65be949f913a" exitCode=0
Jul 10 18:42:11.247374 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:11.246924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc" event={"ID":"ac31c4b7-c733-47fd-8c91-8db7c161c319","Type":"ContainerDied","Data":"f8ef6dd069f409805c8a85819e57114512d51a729e4f54d36eaa65be949f913a"}
Jul 10 18:42:11.247374 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:11.247296    2561 scope.go:117] "RemoveContainer" containerID="f8ef6dd069f409805c8a85819e57114512d51a729e4f54d36eaa65be949f913a"
Jul 10 18:42:11.540577 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:11.540533    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612b8f12-28aa-444e-beb3-5700ebd338f3" path="/var/lib/kubelet/pods/612b8f12-28aa-444e-beb3-5700ebd338f3/volumes"
Jul 10 18:42:12.251848 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:12.251811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n8zbc" event={"ID":"ac31c4b7-c733-47fd-8c91-8db7c161c319","Type":"ContainerStarted","Data":"a6222b45e7a8fb621cd853f2159cc49f2bed2a1d0b28ece780ab4086155a5e94"}
Jul 10 18:42:26.304539 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:26.304502    2561 generic.go:356] "Generic (PLEG): container finished" podID="d91b43b9-931d-417d-bc0b-9609ead67393" containerID="2454f36e3b9b82d6bc930b9d76c73a842c1946da0d8b5759082122b12f64390f" exitCode=0
Jul 10 18:42:26.304928 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:26.304549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j" event={"ID":"d91b43b9-931d-417d-bc0b-9609ead67393","Type":"ContainerDied","Data":"2454f36e3b9b82d6bc930b9d76c73a842c1946da0d8b5759082122b12f64390f"}
Jul 10 18:42:26.304928 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:26.304853    2561 scope.go:117] "RemoveContainer" containerID="2454f36e3b9b82d6bc930b9d76c73a842c1946da0d8b5759082122b12f64390f"
Jul 10 18:42:27.309883 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:42:27.309852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4dz8j" event={"ID":"d91b43b9-931d-417d-bc0b-9609ead67393","Type":"ContainerStarted","Data":"df58aba65a395f3ff7c02f65a0386e7914fc842c6baef21c3f90bd2046dd18cf"}
Jul 10 18:44:53.461988 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:44:53.461959    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:44:53.463745 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:44:53.463721    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:44:53.472924 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:44:53.472900    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 18:46:04.064674 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.064591    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"]
Jul 10 18:46:04.065130 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.064886    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612b8f12-28aa-444e-beb3-5700ebd338f3" containerName="registry"
Jul 10 18:46:04.065130 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.064900    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="612b8f12-28aa-444e-beb3-5700ebd338f3" containerName="registry"
Jul 10 18:46:04.065130 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.064948    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="612b8f12-28aa-444e-beb3-5700ebd338f3" containerName="registry"
Jul 10 18:46:04.067774 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.067756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.078325 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.078298    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tszbt\""
Jul 10 18:46:04.078605 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.078587    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 10 18:46:04.079446 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.079430    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 10 18:46:04.089432 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.089399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.089579 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.089443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.089579 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.089497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2h7\" (UniqueName: \"kubernetes.io/projected/87aa0ad6-8249-45de-be59-133a566489b4-kube-api-access-bj2h7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.093349 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.093309    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"]
Jul 10 18:46:04.190518 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.190476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.190518 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.190524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.190781 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.190552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2h7\" (UniqueName: \"kubernetes.io/projected/87aa0ad6-8249-45de-be59-133a566489b4-kube-api-access-bj2h7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.190887 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.190863    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.190983 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.190963    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.201767 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.201725    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2h7\" (UniqueName: \"kubernetes.io/projected/87aa0ad6-8249-45de-be59-133a566489b4-kube-api-access-bj2h7\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.376998 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.376904    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:04.510552 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.510512    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"]
Jul 10 18:46:04.513836 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:46:04.513807    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87aa0ad6_8249_45de_be59_133a566489b4.slice/crio-993e1171f229e7d714c3b6f1b5a852f1844f3c803ac6750105951aa79ef14547 WatchSource:0}: Error finding container 993e1171f229e7d714c3b6f1b5a852f1844f3c803ac6750105951aa79ef14547: Status 404 returned error can't find the container with id 993e1171f229e7d714c3b6f1b5a852f1844f3c803ac6750105951aa79ef14547
Jul 10 18:46:04.516220 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:04.516206    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:46:05.064771 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:05.064736    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg" event={"ID":"87aa0ad6-8249-45de-be59-133a566489b4","Type":"ContainerStarted","Data":"993e1171f229e7d714c3b6f1b5a852f1844f3c803ac6750105951aa79ef14547"}
Jul 10 18:46:11.088462 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:11.088424    2561 generic.go:356] "Generic (PLEG): container finished" podID="87aa0ad6-8249-45de-be59-133a566489b4" containerID="9caa4ed32ce05a5928c44563056b0a160668cede1a8790a614e876bdc6e637b0" exitCode=0
Jul 10 18:46:11.088939 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:11.088469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg" event={"ID":"87aa0ad6-8249-45de-be59-133a566489b4","Type":"ContainerDied","Data":"9caa4ed32ce05a5928c44563056b0a160668cede1a8790a614e876bdc6e637b0"}
Jul 10 18:46:14.102567 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:14.102533    2561 generic.go:356] "Generic (PLEG): container finished" podID="87aa0ad6-8249-45de-be59-133a566489b4" containerID="a7129b83a84d42e415a4057dabbab935034a5d456bf86c5d371f1a25e5fecd01" exitCode=0
Jul 10 18:46:14.102951 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:14.102628    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg" event={"ID":"87aa0ad6-8249-45de-be59-133a566489b4","Type":"ContainerDied","Data":"a7129b83a84d42e415a4057dabbab935034a5d456bf86c5d371f1a25e5fecd01"}
Jul 10 18:46:22.135597 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:22.135556    2561 generic.go:356] "Generic (PLEG): container finished" podID="87aa0ad6-8249-45de-be59-133a566489b4" containerID="b9ec6806565af93ed630d11c408643a94137536f2860d2b2230a2f107b083682" exitCode=0
Jul 10 18:46:22.135984 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:22.135608    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg" event={"ID":"87aa0ad6-8249-45de-be59-133a566489b4","Type":"ContainerDied","Data":"b9ec6806565af93ed630d11c408643a94137536f2860d2b2230a2f107b083682"}
Jul 10 18:46:23.272860 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.272825    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:23.343012 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.342965    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj2h7\" (UniqueName: \"kubernetes.io/projected/87aa0ad6-8249-45de-be59-133a566489b4-kube-api-access-bj2h7\") pod \"87aa0ad6-8249-45de-be59-133a566489b4\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") "
Jul 10 18:46:23.343224 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.343033    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-util\") pod \"87aa0ad6-8249-45de-be59-133a566489b4\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") "
Jul 10 18:46:23.343224 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.343067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-bundle\") pod \"87aa0ad6-8249-45de-be59-133a566489b4\" (UID: \"87aa0ad6-8249-45de-be59-133a566489b4\") "
Jul 10 18:46:23.343716 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.343685    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-bundle" (OuterVolumeSpecName: "bundle") pod "87aa0ad6-8249-45de-be59-133a566489b4" (UID: "87aa0ad6-8249-45de-be59-133a566489b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:46:23.345319 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.345291    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aa0ad6-8249-45de-be59-133a566489b4-kube-api-access-bj2h7" (OuterVolumeSpecName: "kube-api-access-bj2h7") pod "87aa0ad6-8249-45de-be59-133a566489b4" (UID: "87aa0ad6-8249-45de-be59-133a566489b4"). InnerVolumeSpecName "kube-api-access-bj2h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:46:23.347659 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.347633    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-util" (OuterVolumeSpecName: "util") pod "87aa0ad6-8249-45de-be59-133a566489b4" (UID: "87aa0ad6-8249-45de-be59-133a566489b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:46:23.444531 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.444496    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-bundle\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:46:23.444531 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.444527    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2h7\" (UniqueName: \"kubernetes.io/projected/87aa0ad6-8249-45de-be59-133a566489b4-kube-api-access-bj2h7\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:46:23.444531 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:23.444537    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87aa0ad6-8249-45de-be59-133a566489b4-util\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:46:24.144318 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:24.144286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg" event={"ID":"87aa0ad6-8249-45de-be59-133a566489b4","Type":"ContainerDied","Data":"993e1171f229e7d714c3b6f1b5a852f1844f3c803ac6750105951aa79ef14547"}
Jul 10 18:46:24.144486 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:24.144334    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993e1171f229e7d714c3b6f1b5a852f1844f3c803ac6750105951aa79ef14547"
Jul 10 18:46:24.144486 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:24.144333    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589h55zg"
Jul 10 18:46:27.819044 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819008    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"]
Jul 10 18:46:27.819426 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819268    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87aa0ad6-8249-45de-be59-133a566489b4" containerName="util"
Jul 10 18:46:27.819426 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819279    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87aa0ad6-8249-45de-be59-133a566489b4" containerName="util"
Jul 10 18:46:27.819426 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819287    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87aa0ad6-8249-45de-be59-133a566489b4" containerName="pull"
Jul 10 18:46:27.819426 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819293    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87aa0ad6-8249-45de-be59-133a566489b4" containerName="pull"
Jul 10 18:46:27.819426 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819310    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87aa0ad6-8249-45de-be59-133a566489b4" containerName="extract"
Jul 10 18:46:27.819426 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819316    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87aa0ad6-8249-45de-be59-133a566489b4" containerName="extract"
Jul 10 18:46:27.819426 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.819357    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87aa0ad6-8249-45de-be59-133a566489b4" containerName="extract"
Jul 10 18:46:27.863885 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.863844    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"]
Jul 10 18:46:27.864081 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.863990    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:27.867688 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.867653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 10 18:46:27.867882 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.867779    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-v8xc7\""
Jul 10 18:46:27.867960 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.867912    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 10 18:46:27.868468 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.868449    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 10 18:46:27.975582 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.975552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfpb\" (UniqueName: \"kubernetes.io/projected/0c6f1326-141d-4982-9771-2724bfb4dd18-kube-api-access-xjfpb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6\" (UID: \"0c6f1326-141d-4982-9771-2724bfb4dd18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:27.975733 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:27.975604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0c6f1326-141d-4982-9771-2724bfb4dd18-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6\" (UID: \"0c6f1326-141d-4982-9771-2724bfb4dd18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:28.076547 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:28.076455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0c6f1326-141d-4982-9771-2724bfb4dd18-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6\" (UID: \"0c6f1326-141d-4982-9771-2724bfb4dd18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:28.076691 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:28.076626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfpb\" (UniqueName: \"kubernetes.io/projected/0c6f1326-141d-4982-9771-2724bfb4dd18-kube-api-access-xjfpb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6\" (UID: \"0c6f1326-141d-4982-9771-2724bfb4dd18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:28.078873 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:28.078840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0c6f1326-141d-4982-9771-2724bfb4dd18-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6\" (UID: \"0c6f1326-141d-4982-9771-2724bfb4dd18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:28.097335 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:28.097304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfpb\" (UniqueName: \"kubernetes.io/projected/0c6f1326-141d-4982-9771-2724bfb4dd18-kube-api-access-xjfpb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6\" (UID: \"0c6f1326-141d-4982-9771-2724bfb4dd18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:28.173753 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:28.173714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:28.349012 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:28.348925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"]
Jul 10 18:46:28.352727 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:46:28.352699    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6f1326_141d_4982_9771_2724bfb4dd18.slice/crio-cc7d6f64de31a07684ea74e73e53a628b6cce03aab05664666a4624749411692 WatchSource:0}: Error finding container cc7d6f64de31a07684ea74e73e53a628b6cce03aab05664666a4624749411692: Status 404 returned error can't find the container with id cc7d6f64de31a07684ea74e73e53a628b6cce03aab05664666a4624749411692
Jul 10 18:46:29.163516 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:29.163480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6" event={"ID":"0c6f1326-141d-4982-9771-2724bfb4dd18","Type":"ContainerStarted","Data":"cc7d6f64de31a07684ea74e73e53a628b6cce03aab05664666a4624749411692"}
Jul 10 18:46:31.173768 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:31.173732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6" event={"ID":"0c6f1326-141d-4982-9771-2724bfb4dd18","Type":"ContainerStarted","Data":"87a16522179bc4a5a699c838ae0315d45ffc1fa044956cf20a74d413a53f3c40"}
Jul 10 18:46:31.174238 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:31.173824    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:31.197947 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:31.197887    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6" podStartSLOduration=1.5056916839999999 podStartE2EDuration="4.197870234s" totalImagesPullingTime="2.69217855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:46:31.195462504 +0000 UTC m=+398.149750589" watchObservedRunningTime="2026-07-10 18:46:31.197870234 +0000 UTC m=+398.152158300"
Jul 10 18:46:32.106516 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.106473    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"]
Jul 10 18:46:32.108474 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.108456    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.112003 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.111972    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 10 18:46:32.112179 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.112024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 10 18:46:32.113407 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.113383    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ns72c\""
Jul 10 18:46:32.120614 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.120583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"]
Jul 10 18:46:32.211089 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.211044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htk8d\" (UniqueName: \"kubernetes.io/projected/c1143fc0-070f-46f2-afef-279bf9c739b9-kube-api-access-htk8d\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.211516 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.211124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c1143fc0-070f-46f2-afef-279bf9c739b9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.211516 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.211219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c1143fc0-070f-46f2-afef-279bf9c739b9-certificates\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.302970 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.302932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-s6j7d"]
Jul 10 18:46:32.305365 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.305342    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:32.308158 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.308112    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 10 18:46:32.311709 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.311677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c1143fc0-070f-46f2-afef-279bf9c739b9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.311868 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.311784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c1143fc0-070f-46f2-afef-279bf9c739b9-certificates\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.311868 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.311849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htk8d\" (UniqueName: \"kubernetes.io/projected/c1143fc0-070f-46f2-afef-279bf9c739b9-kube-api-access-htk8d\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.312384 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.312359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c1143fc0-070f-46f2-afef-279bf9c739b9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.316766 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.316732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c1143fc0-070f-46f2-afef-279bf9c739b9-certificates\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.318116 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.318049    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-s6j7d"]
Jul 10 18:46:32.328596 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.328550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htk8d\" (UniqueName: \"kubernetes.io/projected/c1143fc0-070f-46f2-afef-279bf9c739b9-kube-api-access-htk8d\") pod \"keda-metrics-apiserver-559f75f947-jpjcj\" (UID: \"c1143fc0-070f-46f2-afef-279bf9c739b9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.412867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.412821    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-certificates\") pod \"keda-admission-5fbd5c4644-s6j7d\" (UID: \"d1c5f8be-4fb3-4993-b235-d4a22712841e\") " pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:32.413068 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.412884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4bv\" (UniqueName: \"kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-kube-api-access-lw4bv\") pod \"keda-admission-5fbd5c4644-s6j7d\" (UID: \"d1c5f8be-4fb3-4993-b235-d4a22712841e\") " pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:32.422802 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.422763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:32.514368 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.514283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-certificates\") pod \"keda-admission-5fbd5c4644-s6j7d\" (UID: \"d1c5f8be-4fb3-4993-b235-d4a22712841e\") " pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:32.514552 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.514403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4bv\" (UniqueName: \"kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-kube-api-access-lw4bv\") pod \"keda-admission-5fbd5c4644-s6j7d\" (UID: \"d1c5f8be-4fb3-4993-b235-d4a22712841e\") " pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:32.514927 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:46:32.514902    2561 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 10 18:46:32.515017 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:46:32.514939    2561 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-s6j7d: secret "keda-admission-webhooks-certs" not found
Jul 10 18:46:32.515017 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:46:32.514999    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-certificates podName:d1c5f8be-4fb3-4993-b235-d4a22712841e nodeName:}" failed. No retries permitted until 2026-07-10 18:46:33.014978437 +0000 UTC m=+399.969266482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-certificates") pod "keda-admission-5fbd5c4644-s6j7d" (UID: "d1c5f8be-4fb3-4993-b235-d4a22712841e") : secret "keda-admission-webhooks-certs" not found
Jul 10 18:46:32.527576 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.527518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4bv\" (UniqueName: \"kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-kube-api-access-lw4bv\") pod \"keda-admission-5fbd5c4644-s6j7d\" (UID: \"d1c5f8be-4fb3-4993-b235-d4a22712841e\") " pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:32.573654 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:32.573595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"]
Jul 10 18:46:32.577578 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:46:32.577549    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1143fc0_070f_46f2_afef_279bf9c739b9.slice/crio-d8ad2e54eb9cd27faffae59e32e528cbca6ac9a2bf58954edcce68539b20e723 WatchSource:0}: Error finding container d8ad2e54eb9cd27faffae59e32e528cbca6ac9a2bf58954edcce68539b20e723: Status 404 returned error can't find the container with id d8ad2e54eb9cd27faffae59e32e528cbca6ac9a2bf58954edcce68539b20e723
Jul 10 18:46:33.019044 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:33.018990    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-certificates\") pod \"keda-admission-5fbd5c4644-s6j7d\" (UID: \"d1c5f8be-4fb3-4993-b235-d4a22712841e\") " pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:33.021759 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:33.021726    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c5f8be-4fb3-4993-b235-d4a22712841e-certificates\") pod \"keda-admission-5fbd5c4644-s6j7d\" (UID: \"d1c5f8be-4fb3-4993-b235-d4a22712841e\") " pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:33.181671 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:33.181626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj" event={"ID":"c1143fc0-070f-46f2-afef-279bf9c739b9","Type":"ContainerStarted","Data":"d8ad2e54eb9cd27faffae59e32e528cbca6ac9a2bf58954edcce68539b20e723"}
Jul 10 18:46:33.217818 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:33.217778    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:33.354200 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:33.354166    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-s6j7d"]
Jul 10 18:46:33.357281 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:46:33.357253    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c5f8be_4fb3_4993_b235_d4a22712841e.slice/crio-d6935394751bf84ab7369c369912cdbbd0efdf7b41f77b9eb6a0f3b54f6a3143 WatchSource:0}: Error finding container d6935394751bf84ab7369c369912cdbbd0efdf7b41f77b9eb6a0f3b54f6a3143: Status 404 returned error can't find the container with id d6935394751bf84ab7369c369912cdbbd0efdf7b41f77b9eb6a0f3b54f6a3143
Jul 10 18:46:34.187802 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:34.187756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d" event={"ID":"d1c5f8be-4fb3-4993-b235-d4a22712841e","Type":"ContainerStarted","Data":"d6935394751bf84ab7369c369912cdbbd0efdf7b41f77b9eb6a0f3b54f6a3143"}
Jul 10 18:46:36.198039 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:36.197997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj" event={"ID":"c1143fc0-070f-46f2-afef-279bf9c739b9","Type":"ContainerStarted","Data":"d301baa0b5d083e08c723fa29852b35feda5cd76ec692fdcdf1b400219bd7018"}
Jul 10 18:46:36.198535 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:36.198177    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:36.199517 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:36.199495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d" event={"ID":"d1c5f8be-4fb3-4993-b235-d4a22712841e","Type":"ContainerStarted","Data":"6855932c6bef2b5b99108742c6f97658c9bead2af1f5d96e73e589681fcf1cd6"}
Jul 10 18:46:36.199641 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:36.199624    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:46:36.215407 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:36.215353    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj" podStartSLOduration=1.178987634 podStartE2EDuration="4.215335525s" totalImagesPullingTime="3.036347891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:46:36.214138525 +0000 UTC m=+403.168426603" watchObservedRunningTime="2026-07-10 18:46:36.215335525 +0000 UTC m=+403.169623628"
Jul 10 18:46:36.231215 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:36.231159    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d" podStartSLOduration=1.9773443149999999 podStartE2EDuration="4.231127376s" totalImagesPullingTime="2.253783061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:46:36.229810545 +0000 UTC m=+403.184098614" watchObservedRunningTime="2026-07-10 18:46:36.231127376 +0000 UTC m=+403.185415443"
Jul 10 18:46:47.208074 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:47.208035    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-jpjcj"
Jul 10 18:46:52.180005 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:52.179957    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-j4xr6"
Jul 10 18:46:57.205583 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:46:57.205547    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-s6j7d"
Jul 10 18:47:38.104055 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.104014    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-659578d74d-ltmx8"]
Jul 10 18:47:38.106716 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.106688    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.110425 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.110401    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 18:47:38.112081 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.112059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 18:47:38.112290 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.112271    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-cjbd9\""
Jul 10 18:47:38.113368 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.113345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 10 18:47:38.122902 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.122870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-ltmx8"]
Jul 10 18:47:38.176983 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.176943    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-nqbd7"]
Jul 10 18:47:38.179735 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.179712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.183955 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.183919    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 18:47:38.187633 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.187610    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bqnpr\""
Jul 10 18:47:38.196982 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.196947    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nqbd7"]
Jul 10 18:47:38.212008 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.211969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpbn\" (UniqueName: \"kubernetes.io/projected/4beb8329-e0c3-48a0-b3a9-3ece6de9ae08-kube-api-access-ffpbn\") pod \"seaweedfs-64568bcd49-nqbd7\" (UID: \"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08\") " pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.212199 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.212025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4beb8329-e0c3-48a0-b3a9-3ece6de9ae08-data\") pod \"seaweedfs-64568bcd49-nqbd7\" (UID: \"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08\") " pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.212199 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.212120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26qb\" (UniqueName: \"kubernetes.io/projected/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-kube-api-access-b26qb\") pod \"kserve-controller-manager-659578d74d-ltmx8\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") " pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.212199 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.212168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-cert\") pod \"kserve-controller-manager-659578d74d-ltmx8\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") " pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.313225 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.313187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpbn\" (UniqueName: \"kubernetes.io/projected/4beb8329-e0c3-48a0-b3a9-3ece6de9ae08-kube-api-access-ffpbn\") pod \"seaweedfs-64568bcd49-nqbd7\" (UID: \"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08\") " pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.313418 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.313237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4beb8329-e0c3-48a0-b3a9-3ece6de9ae08-data\") pod \"seaweedfs-64568bcd49-nqbd7\" (UID: \"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08\") " pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.313418 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.313290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26qb\" (UniqueName: \"kubernetes.io/projected/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-kube-api-access-b26qb\") pod \"kserve-controller-manager-659578d74d-ltmx8\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") " pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.313418 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.313308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-cert\") pod \"kserve-controller-manager-659578d74d-ltmx8\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") " pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.313777 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.313754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4beb8329-e0c3-48a0-b3a9-3ece6de9ae08-data\") pod \"seaweedfs-64568bcd49-nqbd7\" (UID: \"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08\") " pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.315993 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.315964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-cert\") pod \"kserve-controller-manager-659578d74d-ltmx8\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") " pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.336439 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.336399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpbn\" (UniqueName: \"kubernetes.io/projected/4beb8329-e0c3-48a0-b3a9-3ece6de9ae08-kube-api-access-ffpbn\") pod \"seaweedfs-64568bcd49-nqbd7\" (UID: \"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08\") " pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.336705 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.336685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26qb\" (UniqueName: \"kubernetes.io/projected/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-kube-api-access-b26qb\") pod \"kserve-controller-manager-659578d74d-ltmx8\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") " pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.417821 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.417782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:38.490547 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.490518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:38.560345 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.560308    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-ltmx8"]
Jul 10 18:47:38.563404 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:47:38.563359    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6efbae_988f_40d3_98f0_a27d3ada3e6b.slice/crio-f83ff17079a65ee70dff0f6dc2018d44190ea95515c04d240a6248c304f57a6a WatchSource:0}: Error finding container f83ff17079a65ee70dff0f6dc2018d44190ea95515c04d240a6248c304f57a6a: Status 404 returned error can't find the container with id f83ff17079a65ee70dff0f6dc2018d44190ea95515c04d240a6248c304f57a6a
Jul 10 18:47:38.637043 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:38.637007    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-nqbd7"]
Jul 10 18:47:38.640462 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:47:38.640431    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4beb8329_e0c3_48a0_b3a9_3ece6de9ae08.slice/crio-1470d4f03061cc04f7d751c208ce3aad9157a8bfa53c3b357db44bbeee0dff78 WatchSource:0}: Error finding container 1470d4f03061cc04f7d751c208ce3aad9157a8bfa53c3b357db44bbeee0dff78: Status 404 returned error can't find the container with id 1470d4f03061cc04f7d751c208ce3aad9157a8bfa53c3b357db44bbeee0dff78
Jul 10 18:47:39.447614 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:39.447574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nqbd7" event={"ID":"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08","Type":"ContainerStarted","Data":"1470d4f03061cc04f7d751c208ce3aad9157a8bfa53c3b357db44bbeee0dff78"}
Jul 10 18:47:39.449988 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:39.449917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-ltmx8" event={"ID":"9e6efbae-988f-40d3-98f0-a27d3ada3e6b","Type":"ContainerStarted","Data":"f83ff17079a65ee70dff0f6dc2018d44190ea95515c04d240a6248c304f57a6a"}
Jul 10 18:47:41.462396 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:41.462350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-ltmx8" event={"ID":"9e6efbae-988f-40d3-98f0-a27d3ada3e6b","Type":"ContainerStarted","Data":"696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079"}
Jul 10 18:47:41.462841 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:41.462459    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:47:41.501281 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:41.501212    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-659578d74d-ltmx8" podStartSLOduration=0.729840985 podStartE2EDuration="3.501190759s" totalImagesPullingTime="2.771349774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:41.500508148 +0000 UTC m=+468.454796214" watchObservedRunningTime="2026-07-10 18:47:41.501190759 +0000 UTC m=+468.455478825"
Jul 10 18:47:42.467804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:42.467758    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-nqbd7" event={"ID":"4beb8329-e0c3-48a0-b3a9-3ece6de9ae08","Type":"ContainerStarted","Data":"a89b4090f3a9a38fde302f85d5cf72976dcf89579b39bcc39e750ff9dbab6871"}
Jul 10 18:47:42.468198 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:42.467850    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:47:42.485632 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:42.485564    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-nqbd7" podStartSLOduration=0.741242588 podStartE2EDuration="4.485545394s" totalImagesPullingTime="3.744302806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:42.484853871 +0000 UTC m=+469.439141937" watchObservedRunningTime="2026-07-10 18:47:42.485545394 +0000 UTC m=+469.439833480"
Jul 10 18:47:48.474830 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:47:48.474797    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-nqbd7"
Jul 10 18:48:12.473466 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:12.473429    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:48:13.258524 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.258482    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-ltmx8"]
Jul 10 18:48:13.258742 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.258702    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-659578d74d-ltmx8" podUID="9e6efbae-988f-40d3-98f0-a27d3ada3e6b" containerName="manager" containerID="cri-o://696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079" gracePeriod=10
Jul 10 18:48:13.283954 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.283909    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-659578d74d-grz59"]
Jul 10 18:48:13.286021 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.286002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.295665 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.295629    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-grz59"]
Jul 10 18:48:13.399327 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.399277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7x57\" (UniqueName: \"kubernetes.io/projected/3170a30b-51bc-486b-81bb-a40ff980832c-kube-api-access-d7x57\") pod \"kserve-controller-manager-659578d74d-grz59\" (UID: \"3170a30b-51bc-486b-81bb-a40ff980832c\") " pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.399502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.399347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3170a30b-51bc-486b-81bb-a40ff980832c-cert\") pod \"kserve-controller-manager-659578d74d-grz59\" (UID: \"3170a30b-51bc-486b-81bb-a40ff980832c\") " pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.499917 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.499885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3170a30b-51bc-486b-81bb-a40ff980832c-cert\") pod \"kserve-controller-manager-659578d74d-grz59\" (UID: \"3170a30b-51bc-486b-81bb-a40ff980832c\") " pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.500289 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.499988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7x57\" (UniqueName: \"kubernetes.io/projected/3170a30b-51bc-486b-81bb-a40ff980832c-kube-api-access-d7x57\") pod \"kserve-controller-manager-659578d74d-grz59\" (UID: \"3170a30b-51bc-486b-81bb-a40ff980832c\") " pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.502786 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.502729    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3170a30b-51bc-486b-81bb-a40ff980832c-cert\") pod \"kserve-controller-manager-659578d74d-grz59\" (UID: \"3170a30b-51bc-486b-81bb-a40ff980832c\") " pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.505766 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.505732    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:48:13.508817 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.508744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7x57\" (UniqueName: \"kubernetes.io/projected/3170a30b-51bc-486b-81bb-a40ff980832c-kube-api-access-d7x57\") pod \"kserve-controller-manager-659578d74d-grz59\" (UID: \"3170a30b-51bc-486b-81bb-a40ff980832c\") " pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.600414 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.600357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b26qb\" (UniqueName: \"kubernetes.io/projected/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-kube-api-access-b26qb\") pod \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") "
Jul 10 18:48:13.600600 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.600450    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-cert\") pod \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\" (UID: \"9e6efbae-988f-40d3-98f0-a27d3ada3e6b\") "
Jul 10 18:48:13.602524 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.602496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-cert" (OuterVolumeSpecName: "cert") pod "9e6efbae-988f-40d3-98f0-a27d3ada3e6b" (UID: "9e6efbae-988f-40d3-98f0-a27d3ada3e6b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:48:13.602585 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.602539    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-kube-api-access-b26qb" (OuterVolumeSpecName: "kube-api-access-b26qb") pod "9e6efbae-988f-40d3-98f0-a27d3ada3e6b" (UID: "9e6efbae-988f-40d3-98f0-a27d3ada3e6b"). InnerVolumeSpecName "kube-api-access-b26qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:48:13.608129 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.608097    2561 generic.go:356] "Generic (PLEG): container finished" podID="9e6efbae-988f-40d3-98f0-a27d3ada3e6b" containerID="696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079" exitCode=0
Jul 10 18:48:13.608277 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.608137    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-ltmx8"
Jul 10 18:48:13.608277 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.608175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-ltmx8" event={"ID":"9e6efbae-988f-40d3-98f0-a27d3ada3e6b","Type":"ContainerDied","Data":"696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079"}
Jul 10 18:48:13.608277 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.608225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-ltmx8" event={"ID":"9e6efbae-988f-40d3-98f0-a27d3ada3e6b","Type":"ContainerDied","Data":"f83ff17079a65ee70dff0f6dc2018d44190ea95515c04d240a6248c304f57a6a"}
Jul 10 18:48:13.608277 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.608249    2561 scope.go:117] "RemoveContainer" containerID="696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079"
Jul 10 18:48:13.616786 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.616767    2561 scope.go:117] "RemoveContainer" containerID="696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079"
Jul 10 18:48:13.617082 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:48:13.617064    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079\": container with ID starting with 696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079 not found: ID does not exist" containerID="696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079"
Jul 10 18:48:13.617119 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.617091    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079"} err="failed to get container status \"696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079\": rpc error: code = NotFound desc = could not find container \"696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079\": container with ID starting with 696727a4d6e3756033d5429f4927e1b27d5125b720a9a5b9e46700d493326079 not found: ID does not exist"
Jul 10 18:48:13.629609 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.629573    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-ltmx8"]
Jul 10 18:48:13.633651 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.633620    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-ltmx8"]
Jul 10 18:48:13.639681 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.639658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:13.701271 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.701233    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b26qb\" (UniqueName: \"kubernetes.io/projected/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-kube-api-access-b26qb\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:48:13.701271 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.701272    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e6efbae-988f-40d3-98f0-a27d3ada3e6b-cert\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:48:13.775436 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:13.775343    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-grz59"]
Jul 10 18:48:13.778368 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:48:13.778337    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3170a30b_51bc_486b_81bb_a40ff980832c.slice/crio-cf66ae226c4d25fad99da55d6df9ec8318b263870737a3605eb75ca0dad0b6d7 WatchSource:0}: Error finding container cf66ae226c4d25fad99da55d6df9ec8318b263870737a3605eb75ca0dad0b6d7: Status 404 returned error can't find the container with id cf66ae226c4d25fad99da55d6df9ec8318b263870737a3605eb75ca0dad0b6d7
Jul 10 18:48:14.614253 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:14.614162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-grz59" event={"ID":"3170a30b-51bc-486b-81bb-a40ff980832c","Type":"ContainerStarted","Data":"7f406b8f6363dc07e5d9307821be30fee2d1bff11733f64f9ee907f0d8e44a08"}
Jul 10 18:48:14.614253 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:14.614221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-grz59" event={"ID":"3170a30b-51bc-486b-81bb-a40ff980832c","Type":"ContainerStarted","Data":"cf66ae226c4d25fad99da55d6df9ec8318b263870737a3605eb75ca0dad0b6d7"}
Jul 10 18:48:14.614657 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:14.614309    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:48:14.633858 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:14.633802    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-659578d74d-grz59" podStartSLOduration=1.249667246 podStartE2EDuration="1.633781806s" totalImagesPullingTime="384.11456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:14.631955027 +0000 UTC m=+501.586243093" watchObservedRunningTime="2026-07-10 18:48:14.633781806 +0000 UTC m=+501.588069872"
Jul 10 18:48:15.540621 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:15.540580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e6efbae-988f-40d3-98f0-a27d3ada3e6b" path="/var/lib/kubelet/pods/9e6efbae-988f-40d3-98f0-a27d3ada3e6b/volumes"
Jul 10 18:48:45.624533 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:48:45.624455    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-659578d74d-grz59"
Jul 10 18:49:22.822861 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.822823    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"]
Jul 10 18:49:22.823443 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.823130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e6efbae-988f-40d3-98f0-a27d3ada3e6b" containerName="manager"
Jul 10 18:49:22.823443 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.823156    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e6efbae-988f-40d3-98f0-a27d3ada3e6b" containerName="manager"
Jul 10 18:49:22.823443 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.823221    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e6efbae-988f-40d3-98f0-a27d3ada3e6b" containerName="manager"
Jul 10 18:49:22.826549 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.826522    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:22.829277 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.829247    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-bc128-predictor-serving-cert\""
Jul 10 18:49:22.829422 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.829341    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:49:22.829485 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.829420    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\""
Jul 10 18:49:22.829485 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.829462    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kr64x\""
Jul 10 18:49:22.829648 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.829625    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:49:22.838835 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.838801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"]
Jul 10 18:49:22.927499 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.927457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:22.927690 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.927507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4flp\" (UniqueName: \"kubernetes.io/projected/7650eb2c-498d-42d8-b5df-23614a22a871-kube-api-access-q4flp\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:22.927690 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.927544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7650eb2c-498d-42d8-b5df-23614a22a871-isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:22.927690 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:22.927582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7650eb2c-498d-42d8-b5df-23614a22a871-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.028859 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.028815    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.029022 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.028879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4flp\" (UniqueName: \"kubernetes.io/projected/7650eb2c-498d-42d8-b5df-23614a22a871-kube-api-access-q4flp\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.029022 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.028922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7650eb2c-498d-42d8-b5df-23614a22a871-isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.029022 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.028950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7650eb2c-498d-42d8-b5df-23614a22a871-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.029194 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:49:23.029039    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-bc128-predictor-serving-cert" not found
Jul 10 18:49:23.029194 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:49:23.029119    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls podName:7650eb2c-498d-42d8-b5df-23614a22a871 nodeName:}" failed. No retries permitted until 2026-07-10 18:49:23.52909739 +0000 UTC m=+570.483385452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls") pod "isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" (UID: "7650eb2c-498d-42d8-b5df-23614a22a871") : secret "isvc-raw-sklearn-batcher-bc128-predictor-serving-cert" not found
Jul 10 18:49:23.029491 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.029468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7650eb2c-498d-42d8-b5df-23614a22a871-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.029727 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.029702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7650eb2c-498d-42d8-b5df-23614a22a871-isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.037834 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.037801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4flp\" (UniqueName: \"kubernetes.io/projected/7650eb2c-498d-42d8-b5df-23614a22a871-kube-api-access-q4flp\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.534401 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.534363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.537372 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.537335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls\") pod \"isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.737430 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.737382    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:23.879516 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.879478    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"]
Jul 10 18:49:23.882460 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:49:23.882433    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7650eb2c_498d_42d8_b5df_23614a22a871.slice/crio-117932cddcabf7f2d47c64f477eae020eaf3d338d555384d2a1e05c48ee71171 WatchSource:0}: Error finding container 117932cddcabf7f2d47c64f477eae020eaf3d338d555384d2a1e05c48ee71171: Status 404 returned error can't find the container with id 117932cddcabf7f2d47c64f477eae020eaf3d338d555384d2a1e05c48ee71171
Jul 10 18:49:23.897281 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:23.897244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerStarted","Data":"117932cddcabf7f2d47c64f477eae020eaf3d338d555384d2a1e05c48ee71171"}
Jul 10 18:49:27.917176 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:27.917121    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerStarted","Data":"6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456"}
Jul 10 18:49:30.931507 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:30.931468    2561 generic.go:356] "Generic (PLEG): container finished" podID="7650eb2c-498d-42d8-b5df-23614a22a871" containerID="6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456" exitCode=0
Jul 10 18:49:30.931876 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:30.931515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerDied","Data":"6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456"}
Jul 10 18:49:45.009789 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:45.009749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerStarted","Data":"dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163"}
Jul 10 18:49:48.029555 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:48.029520    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerStarted","Data":"ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d"}
Jul 10 18:49:50.041266 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:50.041168    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerStarted","Data":"d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16"}
Jul 10 18:49:50.041660 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:50.041381    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:50.041660 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:50.041438    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:50.041660 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:50.041471    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:50.042808 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:50.042755    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:49:50.043572 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:50.043542    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:50.063162 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:50.063076    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podStartSLOduration=2.647361162 podStartE2EDuration="28.063058483s" totalImagesPullingTime="21.737404711s" totalInitContainerRuntime="3.67829261s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:49:22 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:50.062082724 +0000 UTC m=+597.016370827" watchObservedRunningTime="2026-07-10 18:49:50.063058483 +0000 UTC m=+597.017346550"
Jul 10 18:49:51.045935 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:51.045882    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:49:51.046469 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:51.046441    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:53.493946 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:53.493914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:49:53.494432 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:53.493914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:49:56.050579 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:56.050548    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:49:56.051209 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:56.051184    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:49:56.051537 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:49:56.051506    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:06.051297 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:06.051256    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:50:06.051762 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:06.051736    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:16.052076 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:16.051979    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:50:16.052840 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:16.052809    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:26.051134 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:26.051088    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:50:26.051598 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:26.051493    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:36.051205 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:36.051160    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:50:36.051658 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:36.051633    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:46.051474 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:46.051425    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:50:46.051921 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:46.051672    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:56.051743 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:56.051710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:50:56.052182 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:50:56.051948    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:51:07.852647 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.852498    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"]
Jul 10 18:51:07.853258 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.853020    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" containerID="cri-o://dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163" gracePeriod=30
Jul 10 18:51:07.853850 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.853531    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy" containerID="cri-o://ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d" gracePeriod=30
Jul 10 18:51:07.853850 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.853745    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" containerID="cri-o://d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16" gracePeriod=30
Jul 10 18:51:07.854885 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.854848    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"]
Jul 10 18:51:07.858669 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.858644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:07.861770 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.861743    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c446e-predictor-serving-cert\""
Jul 10 18:51:07.861880 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.861752    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\""
Jul 10 18:51:07.872487 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.872448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"]
Jul 10 18:51:07.919390 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.919354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87bb0cba-50b1-4c39-b042-aebe1bb0076b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:07.919581 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.919418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87bb0cba-50b1-4c39-b042-aebe1bb0076b-isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:07.919581 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.919480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:07.919581 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:07.919513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk488\" (UniqueName: \"kubernetes.io/projected/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kube-api-access-nk488\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.020372 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.020324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87bb0cba-50b1-4c39-b042-aebe1bb0076b-isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.020372 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.020374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.020627 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.020401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk488\" (UniqueName: \"kubernetes.io/projected/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kube-api-access-nk488\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.020627 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.020430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87bb0cba-50b1-4c39-b042-aebe1bb0076b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.020881 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.020859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.021133 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.021114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87bb0cba-50b1-4c39-b042-aebe1bb0076b-isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.022968 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.022947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87bb0cba-50b1-4c39-b042-aebe1bb0076b-proxy-tls\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.030080 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.030038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk488\" (UniqueName: \"kubernetes.io/projected/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kube-api-access-nk488\") pod \"isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.180020 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.179981    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:08.325264 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.325232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"]
Jul 10 18:51:08.328206 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:51:08.328174    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bb0cba_50b1_4c39_b042_aebe1bb0076b.slice/crio-160e5a84d282c92a26dfe5f24a5c1331b67ec53940fa03b5899f0a8846630bf8 WatchSource:0}: Error finding container 160e5a84d282c92a26dfe5f24a5c1331b67ec53940fa03b5899f0a8846630bf8: Status 404 returned error can't find the container with id 160e5a84d282c92a26dfe5f24a5c1331b67ec53940fa03b5899f0a8846630bf8
Jul 10 18:51:08.385852 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.385816    2561 generic.go:356] "Generic (PLEG): container finished" podID="7650eb2c-498d-42d8-b5df-23614a22a871" containerID="ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d" exitCode=2
Jul 10 18:51:08.386175 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.385880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerDied","Data":"ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d"}
Jul 10 18:51:08.387503 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.387479    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerStarted","Data":"fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7"}
Jul 10 18:51:08.387617 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:08.387529    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerStarted","Data":"160e5a84d282c92a26dfe5f24a5c1331b67ec53940fa03b5899f0a8846630bf8"}
Jul 10 18:51:11.046170 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:11.046100    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 10 18:51:13.413122 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:13.413088    2561 generic.go:356] "Generic (PLEG): container finished" podID="7650eb2c-498d-42d8-b5df-23614a22a871" containerID="dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163" exitCode=0
Jul 10 18:51:13.413541 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:13.413180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerDied","Data":"dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163"}
Jul 10 18:51:13.414563 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:13.414539    2561 generic.go:356] "Generic (PLEG): container finished" podID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerID="fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7" exitCode=0
Jul 10 18:51:13.414667 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:13.414581    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerDied","Data":"fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7"}
Jul 10 18:51:14.421814 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:14.421775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerStarted","Data":"d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1"}
Jul 10 18:51:14.422196 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:14.421846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerStarted","Data":"ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866"}
Jul 10 18:51:14.422196 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:14.422046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:14.443842 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:14.443781    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podStartSLOduration=3.41889752 podStartE2EDuration="7.443763999s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.024866479s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:14.441799867 +0000 UTC m=+681.396087934" watchObservedRunningTime="2026-07-10 18:51:14.443763999 +0000 UTC m=+681.398052065"
Jul 10 18:51:15.426794 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:15.426754    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:15.428119 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:15.428081    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:51:16.046362 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:16.046318    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 10 18:51:16.051626 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:16.051588    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:51:16.051942 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:16.051910    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:16.431312 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:16.431255    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:51:21.046378 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:21.046328    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 10 18:51:21.046771 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:21.046498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:51:21.437460 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:21.437432    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:51:21.438105 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:21.438078    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:51:26.046673 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:26.046618    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 10 18:51:26.052067 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:26.052030    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:51:26.052423 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:26.052396    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:31.046505 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:31.046454    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 10 18:51:31.438619 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:31.438578    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:51:36.046089 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:36.046043    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 10 18:51:36.051444 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:36.051393    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:51:36.051580 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:36.051551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:51:36.051809 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:36.051782    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:36.051914 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:36.051903    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:51:38.010076 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.010044    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:51:38.174110 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.174074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4flp\" (UniqueName: \"kubernetes.io/projected/7650eb2c-498d-42d8-b5df-23614a22a871-kube-api-access-q4flp\") pod \"7650eb2c-498d-42d8-b5df-23614a22a871\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") "
Jul 10 18:51:38.174327 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.174132    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7650eb2c-498d-42d8-b5df-23614a22a871-kserve-provision-location\") pod \"7650eb2c-498d-42d8-b5df-23614a22a871\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") "
Jul 10 18:51:38.174327 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.174179    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls\") pod \"7650eb2c-498d-42d8-b5df-23614a22a871\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") "
Jul 10 18:51:38.174327 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.174248    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7650eb2c-498d-42d8-b5df-23614a22a871-isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\") pod \"7650eb2c-498d-42d8-b5df-23614a22a871\" (UID: \"7650eb2c-498d-42d8-b5df-23614a22a871\") "
Jul 10 18:51:38.174556 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.174515    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650eb2c-498d-42d8-b5df-23614a22a871-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7650eb2c-498d-42d8-b5df-23614a22a871" (UID: "7650eb2c-498d-42d8-b5df-23614a22a871"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:51:38.174720 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.174693    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7650eb2c-498d-42d8-b5df-23614a22a871-isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config") pod "7650eb2c-498d-42d8-b5df-23614a22a871" (UID: "7650eb2c-498d-42d8-b5df-23614a22a871"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:51:38.176425 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.176402    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7650eb2c-498d-42d8-b5df-23614a22a871" (UID: "7650eb2c-498d-42d8-b5df-23614a22a871"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:51:38.176515 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.176457    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650eb2c-498d-42d8-b5df-23614a22a871-kube-api-access-q4flp" (OuterVolumeSpecName: "kube-api-access-q4flp") pod "7650eb2c-498d-42d8-b5df-23614a22a871" (UID: "7650eb2c-498d-42d8-b5df-23614a22a871"). InnerVolumeSpecName "kube-api-access-q4flp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:51:38.274832 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.274775    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7650eb2c-498d-42d8-b5df-23614a22a871-isvc-raw-sklearn-batcher-bc128-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:51:38.274832 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.274825    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4flp\" (UniqueName: \"kubernetes.io/projected/7650eb2c-498d-42d8-b5df-23614a22a871-kube-api-access-q4flp\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:51:38.274832 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.274835    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7650eb2c-498d-42d8-b5df-23614a22a871-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:51:38.274832 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.274844    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7650eb2c-498d-42d8-b5df-23614a22a871-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:51:38.529268 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.529172    2561 generic.go:356] "Generic (PLEG): container finished" podID="7650eb2c-498d-42d8-b5df-23614a22a871" containerID="d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16" exitCode=0
Jul 10 18:51:38.529268 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.529229    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerDied","Data":"d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16"}
Jul 10 18:51:38.529268 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.529240    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"
Jul 10 18:51:38.529468 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.529300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr" event={"ID":"7650eb2c-498d-42d8-b5df-23614a22a871","Type":"ContainerDied","Data":"117932cddcabf7f2d47c64f477eae020eaf3d338d555384d2a1e05c48ee71171"}
Jul 10 18:51:38.529468 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.529346    2561 scope.go:117] "RemoveContainer" containerID="d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16"
Jul 10 18:51:38.537739 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.537717    2561 scope.go:117] "RemoveContainer" containerID="ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d"
Jul 10 18:51:38.545800 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.545776    2561 scope.go:117] "RemoveContainer" containerID="dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163"
Jul 10 18:51:38.551130 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.551104    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"]
Jul 10 18:51:38.554755 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.554734    2561 scope.go:117] "RemoveContainer" containerID="6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456"
Jul 10 18:51:38.555076 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.555052    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bc128-predictor-5c8d8d67d8-ktdvr"]
Jul 10 18:51:38.563089 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.563070    2561 scope.go:117] "RemoveContainer" containerID="d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16"
Jul 10 18:51:38.563446 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:51:38.563425    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16\": container with ID starting with d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16 not found: ID does not exist" containerID="d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16"
Jul 10 18:51:38.563515 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.563460    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16"} err="failed to get container status \"d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16\": rpc error: code = NotFound desc = could not find container \"d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16\": container with ID starting with d91bed1905720fcb7ca69f433a13ef3617638638d6176ea4935211723ad74c16 not found: ID does not exist"
Jul 10 18:51:38.563515 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.563486    2561 scope.go:117] "RemoveContainer" containerID="ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d"
Jul 10 18:51:38.563747 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:51:38.563732    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d\": container with ID starting with ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d not found: ID does not exist" containerID="ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d"
Jul 10 18:51:38.563804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.563754    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d"} err="failed to get container status \"ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d\": rpc error: code = NotFound desc = could not find container \"ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d\": container with ID starting with ace164ea0382439bcde020203a19de46e70b346f2ff5427d44019b639d6e4c3d not found: ID does not exist"
Jul 10 18:51:38.563804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.563774    2561 scope.go:117] "RemoveContainer" containerID="dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163"
Jul 10 18:51:38.564018 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:51:38.564001    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163\": container with ID starting with dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163 not found: ID does not exist" containerID="dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163"
Jul 10 18:51:38.564086 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.564026    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163"} err="failed to get container status \"dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163\": rpc error: code = NotFound desc = could not find container \"dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163\": container with ID starting with dec1aa1eab2ab20880a1cde48fc412906d4773891e32ef870a12a735e0e8a163 not found: ID does not exist"
Jul 10 18:51:38.564086 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.564045    2561 scope.go:117] "RemoveContainer" containerID="6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456"
Jul 10 18:51:38.564373 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:51:38.564354    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456\": container with ID starting with 6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456 not found: ID does not exist" containerID="6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456"
Jul 10 18:51:38.564425 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:38.564379    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456"} err="failed to get container status \"6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456\": rpc error: code = NotFound desc = could not find container \"6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456\": container with ID starting with 6ea15e8f4d63cee638a75cad74ede2314be9c98077eb59f851047560f9da9456 not found: ID does not exist"
Jul 10 18:51:39.540491 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:39.540450    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" path="/var/lib/kubelet/pods/7650eb2c-498d-42d8-b5df-23614a22a871/volumes"
Jul 10 18:51:41.439076 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:41.439029    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:51:51.438536 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:51:51.438430    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:52:01.438805 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:01.438750    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:52:11.438290 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:11.438235    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:52:21.439313 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:21.439279    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:52:37.859242 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859138    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"]
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859530    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container"
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859545    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container"
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859560    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent"
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859567    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent"
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859576    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="storage-initializer"
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="storage-initializer"
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859597    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy"
Jul 10 18:52:37.859620 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859603    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy"
Jul 10 18:52:37.859855 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859659    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kube-rbac-proxy"
Jul 10 18:52:37.859855 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859667    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="agent"
Jul 10 18:52:37.859855 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.859674    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7650eb2c-498d-42d8-b5df-23614a22a871" containerName="kserve-container"
Jul 10 18:52:37.861598 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.861579    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:37.864074 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.864030    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-c446e-serving-cert\""
Jul 10 18:52:37.864074 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.864030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-c446e-kube-rbac-proxy-sar-config\""
Jul 10 18:52:37.871409 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.871380    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"]
Jul 10 18:52:37.936962 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.936918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e32ae0e-27e9-4e83-9754-9e368651ab4a-proxy-tls\") pod \"model-chainer-raw-c446e-55b9c8fd58-nh2z8\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") " pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:37.937132 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:37.936987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e32ae0e-27e9-4e83-9754-9e368651ab4a-openshift-service-ca-bundle\") pod \"model-chainer-raw-c446e-55b9c8fd58-nh2z8\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") " pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:38.037587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.037546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e32ae0e-27e9-4e83-9754-9e368651ab4a-openshift-service-ca-bundle\") pod \"model-chainer-raw-c446e-55b9c8fd58-nh2z8\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") " pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:38.037757 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.037612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e32ae0e-27e9-4e83-9754-9e368651ab4a-proxy-tls\") pod \"model-chainer-raw-c446e-55b9c8fd58-nh2z8\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") " pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:38.038274 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.038233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e32ae0e-27e9-4e83-9754-9e368651ab4a-openshift-service-ca-bundle\") pod \"model-chainer-raw-c446e-55b9c8fd58-nh2z8\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") " pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:38.040155 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.040124    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e32ae0e-27e9-4e83-9754-9e368651ab4a-proxy-tls\") pod \"model-chainer-raw-c446e-55b9c8fd58-nh2z8\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") " pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:38.174628 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.174596    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:38.311264 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.311233    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"]
Jul 10 18:52:38.313942 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:52:38.313913    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e32ae0e_27e9_4e83_9754_9e368651ab4a.slice/crio-4de149aa54692f659a10dd050b86800f73520af1932e6f300e54687ab04e9a0f WatchSource:0}: Error finding container 4de149aa54692f659a10dd050b86800f73520af1932e6f300e54687ab04e9a0f: Status 404 returned error can't find the container with id 4de149aa54692f659a10dd050b86800f73520af1932e6f300e54687ab04e9a0f
Jul 10 18:52:38.315857 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.315841    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:52:38.786763 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:38.786728    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" event={"ID":"3e32ae0e-27e9-4e83-9754-9e368651ab4a","Type":"ContainerStarted","Data":"4de149aa54692f659a10dd050b86800f73520af1932e6f300e54687ab04e9a0f"}
Jul 10 18:52:40.796942 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:40.796900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" event={"ID":"3e32ae0e-27e9-4e83-9754-9e368651ab4a","Type":"ContainerStarted","Data":"cff55a4e21233038908a22c73ebd807e1f1483eab1a597d9728e25f635e7e6c9"}
Jul 10 18:52:40.797369 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:40.796996    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:40.813810 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:40.813741    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podStartSLOduration=1.447860849 podStartE2EDuration="3.813723831s" totalImagesPullingTime="2.365862982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:52:40.812493858 +0000 UTC m=+767.766781928" watchObservedRunningTime="2026-07-10 18:52:40.813723831 +0000 UTC m=+767.768011942"
Jul 10 18:52:46.808393 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:46.808355    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:52:47.935071 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:47.935035    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"]
Jul 10 18:52:47.935593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:47.935232    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e" containerID="cri-o://cff55a4e21233038908a22c73ebd807e1f1483eab1a597d9728e25f635e7e6c9" gracePeriod=30
Jul 10 18:52:48.170249 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.170200    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"]
Jul 10 18:52:48.173332 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.173305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.177705 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.177664    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e9877-predictor-serving-cert\""
Jul 10 18:52:48.177966 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.177679    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\""
Jul 10 18:52:48.179667 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.179636    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"]
Jul 10 18:52:48.179919 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.179890    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" containerID="cri-o://ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866" gracePeriod=30
Jul 10 18:52:48.180054 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.179975    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kube-rbac-proxy" containerID="cri-o://d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1" gracePeriod=30
Jul 10 18:52:48.186236 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.186056    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"]
Jul 10 18:52:48.325642 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.325603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17def5e7-a73d-4836-a10d-a4b8dd37a860-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.325851 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.325693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17def5e7-a73d-4836-a10d-a4b8dd37a860-isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.325851 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.325737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17def5e7-a73d-4836-a10d-a4b8dd37a860-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.325851 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.325806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6s28\" (UniqueName: \"kubernetes.io/projected/17def5e7-a73d-4836-a10d-a4b8dd37a860-kube-api-access-w6s28\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.426491 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.426450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6s28\" (UniqueName: \"kubernetes.io/projected/17def5e7-a73d-4836-a10d-a4b8dd37a860-kube-api-access-w6s28\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.426682 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.426513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17def5e7-a73d-4836-a10d-a4b8dd37a860-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.426682 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.426565    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17def5e7-a73d-4836-a10d-a4b8dd37a860-isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.426682 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.426598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17def5e7-a73d-4836-a10d-a4b8dd37a860-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.426980 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.426958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17def5e7-a73d-4836-a10d-a4b8dd37a860-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.427381 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.427354    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17def5e7-a73d-4836-a10d-a4b8dd37a860-isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.429240 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.429216    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17def5e7-a73d-4836-a10d-a4b8dd37a860-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.435979 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.435947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6s28\" (UniqueName: \"kubernetes.io/projected/17def5e7-a73d-4836-a10d-a4b8dd37a860-kube-api-access-w6s28\") pod \"isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.491945 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.491855    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:48.637691 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.637534    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"]
Jul 10 18:52:48.640543 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:52:48.640517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17def5e7_a73d_4836_a10d_a4b8dd37a860.slice/crio-43622c0ceb60c5c4f80e51b89b1e236c5b5219f4214f84c92742498bbe8f4214 WatchSource:0}: Error finding container 43622c0ceb60c5c4f80e51b89b1e236c5b5219f4214f84c92742498bbe8f4214: Status 404 returned error can't find the container with id 43622c0ceb60c5c4f80e51b89b1e236c5b5219f4214f84c92742498bbe8f4214
Jul 10 18:52:48.835771 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.835670    2561 generic.go:356] "Generic (PLEG): container finished" podID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerID="d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1" exitCode=2
Jul 10 18:52:48.835771 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.835744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerDied","Data":"d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1"}
Jul 10 18:52:48.837390 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.837357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerStarted","Data":"40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20"}
Jul 10 18:52:48.837532 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:48.837414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerStarted","Data":"43622c0ceb60c5c4f80e51b89b1e236c5b5219f4214f84c92742498bbe8f4214"}
Jul 10 18:52:51.431618 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:51.431572    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 10 18:52:51.438381 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:51.438344    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:52:51.804967 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:51.804874    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:52.856666 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:52.856628    2561 generic.go:356] "Generic (PLEG): container finished" podID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerID="40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20" exitCode=0
Jul 10 18:52:52.857063 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:52.856698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerDied","Data":"40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20"}
Jul 10 18:52:53.034401 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.034371    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:52:53.163702 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.163663    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87bb0cba-50b1-4c39-b042-aebe1bb0076b-proxy-tls\") pod \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") "
Jul 10 18:52:53.163877 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.163725    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kserve-provision-location\") pod \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") "
Jul 10 18:52:53.163877 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.163769    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87bb0cba-50b1-4c39-b042-aebe1bb0076b-isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\") pod \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") "
Jul 10 18:52:53.163877 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.163802    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk488\" (UniqueName: \"kubernetes.io/projected/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kube-api-access-nk488\") pod \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\" (UID: \"87bb0cba-50b1-4c39-b042-aebe1bb0076b\") "
Jul 10 18:52:53.164058 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.164025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87bb0cba-50b1-4c39-b042-aebe1bb0076b" (UID: "87bb0cba-50b1-4c39-b042-aebe1bb0076b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:52:53.164164 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.164120    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87bb0cba-50b1-4c39-b042-aebe1bb0076b-isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config") pod "87bb0cba-50b1-4c39-b042-aebe1bb0076b" (UID: "87bb0cba-50b1-4c39-b042-aebe1bb0076b"). InnerVolumeSpecName "isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:52:53.165921 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.165880    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87bb0cba-50b1-4c39-b042-aebe1bb0076b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87bb0cba-50b1-4c39-b042-aebe1bb0076b" (UID: "87bb0cba-50b1-4c39-b042-aebe1bb0076b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:52:53.165921 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.165889    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kube-api-access-nk488" (OuterVolumeSpecName: "kube-api-access-nk488") pod "87bb0cba-50b1-4c39-b042-aebe1bb0076b" (UID: "87bb0cba-50b1-4c39-b042-aebe1bb0076b"). InnerVolumeSpecName "kube-api-access-nk488". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:52:53.264523 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.264468    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk488\" (UniqueName: \"kubernetes.io/projected/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kube-api-access-nk488\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:52:53.264523 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.264515    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87bb0cba-50b1-4c39-b042-aebe1bb0076b-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:52:53.264523 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.264527    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87bb0cba-50b1-4c39-b042-aebe1bb0076b-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:52:53.264523 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.264536    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87bb0cba-50b1-4c39-b042-aebe1bb0076b-isvc-sklearn-graph-raw-c446e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:52:53.863489 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.863395    2561 generic.go:356] "Generic (PLEG): container finished" podID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerID="ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866" exitCode=0
Jul 10 18:52:53.863489 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.863473    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"
Jul 10 18:52:53.863937 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.863482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerDied","Data":"ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866"}
Jul 10 18:52:53.863937 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.863598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q" event={"ID":"87bb0cba-50b1-4c39-b042-aebe1bb0076b","Type":"ContainerDied","Data":"160e5a84d282c92a26dfe5f24a5c1331b67ec53940fa03b5899f0a8846630bf8"}
Jul 10 18:52:53.863937 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.863640    2561 scope.go:117] "RemoveContainer" containerID="d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1"
Jul 10 18:52:53.865926 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.865892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerStarted","Data":"26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4"}
Jul 10 18:52:53.866054 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.865963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerStarted","Data":"b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275"}
Jul 10 18:52:53.866318 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.866301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:53.866520 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.866501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:53.867963 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.867933    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:52:53.872459 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.872436    2561 scope.go:117] "RemoveContainer" containerID="ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866"
Jul 10 18:52:53.880708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.880685    2561 scope.go:117] "RemoveContainer" containerID="fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7"
Jul 10 18:52:53.888641 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.888615    2561 scope.go:117] "RemoveContainer" containerID="d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1"
Jul 10 18:52:53.888984 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:52:53.888965    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1\": container with ID starting with d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1 not found: ID does not exist" containerID="d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1"
Jul 10 18:52:53.889055 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.888993    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1"} err="failed to get container status \"d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1\": rpc error: code = NotFound desc = could not find container \"d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1\": container with ID starting with d4f128e1ab2de6e7008b9d418288a024657b84800b4d469c837bde7b197fbfd1 not found: ID does not exist"
Jul 10 18:52:53.889055 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.889012    2561 scope.go:117] "RemoveContainer" containerID="ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866"
Jul 10 18:52:53.889311 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:52:53.889293    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866\": container with ID starting with ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866 not found: ID does not exist" containerID="ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866"
Jul 10 18:52:53.889355 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.889316    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866"} err="failed to get container status \"ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866\": rpc error: code = NotFound desc = could not find container \"ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866\": container with ID starting with ca2c1a926a6c8dc572e12102b7180812805910c08b1812be26c6807862351866 not found: ID does not exist"
Jul 10 18:52:53.889355 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.889332    2561 scope.go:117] "RemoveContainer" containerID="fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7"
Jul 10 18:52:53.889624 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:52:53.889604    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7\": container with ID starting with fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7 not found: ID does not exist" containerID="fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7"
Jul 10 18:52:53.889696 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.889635    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7"} err="failed to get container status \"fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7\": rpc error: code = NotFound desc = could not find container \"fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7\": container with ID starting with fd96b8065d0db2a91e4fcb7d8b03f3bbf252d4e59f4db3c14c33c490080843a7 not found: ID does not exist"
Jul 10 18:52:53.891948 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.891903    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podStartSLOduration=2.210323246 podStartE2EDuration="5.891889934s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.681566688s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:52:53.890018847 +0000 UTC m=+780.844306927" watchObservedRunningTime="2026-07-10 18:52:53.891889934 +0000 UTC m=+780.846178000"
Jul 10 18:52:53.909755 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.909714    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"]
Jul 10 18:52:53.912747 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:53.912711    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c446e-predictor-7579489dd-wwq8q"]
Jul 10 18:52:54.871354 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:54.871316    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:52:55.540955 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:55.540918    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" path="/var/lib/kubelet/pods/87bb0cba-50b1-4c39-b042-aebe1bb0076b/volumes"
Jul 10 18:52:56.805527 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:56.805482    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:59.876029 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:59.875995    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:52:59.876544 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:52:59.876515    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:53:01.805603 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:01.805561    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:01.805992 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:01.805672    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:53:06.805220 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:06.805178    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:09.876640 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:09.876593    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:53:11.805829 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:11.805784    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:16.804857 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:16.804762    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:17.970576 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:17.970533    2561 generic.go:356] "Generic (PLEG): container finished" podID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerID="cff55a4e21233038908a22c73ebd807e1f1483eab1a597d9728e25f635e7e6c9" exitCode=0
Jul 10 18:53:17.970980 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:17.970597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" event={"ID":"3e32ae0e-27e9-4e83-9754-9e368651ab4a","Type":"ContainerDied","Data":"cff55a4e21233038908a22c73ebd807e1f1483eab1a597d9728e25f635e7e6c9"}
Jul 10 18:53:18.089974 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.089945    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:53:18.167801 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.167759    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e32ae0e-27e9-4e83-9754-9e368651ab4a-openshift-service-ca-bundle\") pod \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") "
Jul 10 18:53:18.167979 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.167843    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e32ae0e-27e9-4e83-9754-9e368651ab4a-proxy-tls\") pod \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\" (UID: \"3e32ae0e-27e9-4e83-9754-9e368651ab4a\") "
Jul 10 18:53:18.168215 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.168187    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e32ae0e-27e9-4e83-9754-9e368651ab4a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3e32ae0e-27e9-4e83-9754-9e368651ab4a" (UID: "3e32ae0e-27e9-4e83-9754-9e368651ab4a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:53:18.170019 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.169983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e32ae0e-27e9-4e83-9754-9e368651ab4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e32ae0e-27e9-4e83-9754-9e368651ab4a" (UID: "3e32ae0e-27e9-4e83-9754-9e368651ab4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:53:18.268386 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.268281    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e32ae0e-27e9-4e83-9754-9e368651ab4a-openshift-service-ca-bundle\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:53:18.268386 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.268314    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e32ae0e-27e9-4e83-9754-9e368651ab4a-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:53:18.975941 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.975912    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"
Jul 10 18:53:18.976379 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.975911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8" event={"ID":"3e32ae0e-27e9-4e83-9754-9e368651ab4a","Type":"ContainerDied","Data":"4de149aa54692f659a10dd050b86800f73520af1932e6f300e54687ab04e9a0f"}
Jul 10 18:53:18.976379 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.976051    2561 scope.go:117] "RemoveContainer" containerID="cff55a4e21233038908a22c73ebd807e1f1483eab1a597d9728e25f635e7e6c9"
Jul 10 18:53:18.997936 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:18.997901    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"]
Jul 10 18:53:19.002068 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:19.002036    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-c446e-55b9c8fd58-nh2z8"]
Jul 10 18:53:19.541283 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:19.541245    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" path="/var/lib/kubelet/pods/3e32ae0e-27e9-4e83-9754-9e368651ab4a/volumes"
Jul 10 18:53:19.877160 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:19.877052    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:53:29.876890 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:29.876846    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:53:39.877131 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:39.877085    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:53:49.877159 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:49.877116    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:53:59.877018 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:53:59.876975    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:54:09.878064 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:09.878024    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:54:28.241076 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241036    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"]
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241367    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241380    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241392    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="storage-initializer"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241397    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="storage-initializer"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241407    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241412    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241423    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kube-rbac-proxy"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241431    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kube-rbac-proxy"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241499    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kserve-container"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241512    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="87bb0cba-50b1-4c39-b042-aebe1bb0076b" containerName="kube-rbac-proxy"
Jul 10 18:54:28.241551 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.241519    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e32ae0e-27e9-4e83-9754-9e368651ab4a" containerName="model-chainer-raw-c446e"
Jul 10 18:54:28.244283 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.244263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.248186 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.248156    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e9877-serving-cert\""
Jul 10 18:54:28.248909 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.248882    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e9877-kube-rbac-proxy-sar-config\""
Jul 10 18:54:28.278804 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.278761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"]
Jul 10 18:54:28.326727 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.326681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eea447-c8b2-4279-8ec6-5c0583dba188-proxy-tls\") pod \"model-chainer-raw-hpa-e9877-646885f67d-vwf6d\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.326727 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.326733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91eea447-c8b2-4279-8ec6-5c0583dba188-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e9877-646885f67d-vwf6d\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.427967 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.427930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eea447-c8b2-4279-8ec6-5c0583dba188-proxy-tls\") pod \"model-chainer-raw-hpa-e9877-646885f67d-vwf6d\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.427967 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.427976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91eea447-c8b2-4279-8ec6-5c0583dba188-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e9877-646885f67d-vwf6d\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.428612 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.428590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91eea447-c8b2-4279-8ec6-5c0583dba188-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e9877-646885f67d-vwf6d\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.430675 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.430650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eea447-c8b2-4279-8ec6-5c0583dba188-proxy-tls\") pod \"model-chainer-raw-hpa-e9877-646885f67d-vwf6d\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.555974 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.555872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:28.708100 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:28.708065    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"]
Jul 10 18:54:28.711591 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:54:28.711563    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91eea447_c8b2_4279_8ec6_5c0583dba188.slice/crio-ee7ab470b8b7cbf1532f89152cccbfc2f991bae94c0eaa8de831eaddc74d9931 WatchSource:0}: Error finding container ee7ab470b8b7cbf1532f89152cccbfc2f991bae94c0eaa8de831eaddc74d9931: Status 404 returned error can't find the container with id ee7ab470b8b7cbf1532f89152cccbfc2f991bae94c0eaa8de831eaddc74d9931
Jul 10 18:54:29.274043 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:29.274004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" event={"ID":"91eea447-c8b2-4279-8ec6-5c0583dba188","Type":"ContainerStarted","Data":"8e3635968d17ade8da50b56b08481b1197a31f28c7719d427d4ad8303127b8f2"}
Jul 10 18:54:29.274478 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:29.274067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" event={"ID":"91eea447-c8b2-4279-8ec6-5c0583dba188","Type":"ContainerStarted","Data":"ee7ab470b8b7cbf1532f89152cccbfc2f991bae94c0eaa8de831eaddc74d9931"}
Jul 10 18:54:29.274478 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:29.274122    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:29.292873 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:29.292818    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podStartSLOduration=1.292799332 podStartE2EDuration="1.292799332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:54:29.291294411 +0000 UTC m=+876.245582492" watchObservedRunningTime="2026-07-10 18:54:29.292799332 +0000 UTC m=+876.247087398"
Jul 10 18:54:35.284950 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:35.284917    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:38.273103 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:38.273064    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"]
Jul 10 18:54:38.273543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:38.273291    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877" containerID="cri-o://8e3635968d17ade8da50b56b08481b1197a31f28c7719d427d4ad8303127b8f2" gracePeriod=30
Jul 10 18:54:38.545935 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:38.545846    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"]
Jul 10 18:54:38.546244 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:38.546186    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" containerID="cri-o://b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275" gracePeriod=30
Jul 10 18:54:38.546489 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:38.546267    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kube-rbac-proxy" containerID="cri-o://26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4" gracePeriod=30
Jul 10 18:54:39.318189 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:39.318133    2561 generic.go:356] "Generic (PLEG): container finished" podID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerID="26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4" exitCode=2
Jul 10 18:54:39.318582 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:39.318208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerDied","Data":"26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4"}
Jul 10 18:54:39.872267 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:39.872219    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 10 18:54:39.876585 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:39.876551    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:54:40.283669 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:40.283624    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:54:43.098988 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.098959    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:54:43.149598 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.149560    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17def5e7-a73d-4836-a10d-a4b8dd37a860-kserve-provision-location\") pod \"17def5e7-a73d-4836-a10d-a4b8dd37a860\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") "
Jul 10 18:54:43.149598 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.149604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6s28\" (UniqueName: \"kubernetes.io/projected/17def5e7-a73d-4836-a10d-a4b8dd37a860-kube-api-access-w6s28\") pod \"17def5e7-a73d-4836-a10d-a4b8dd37a860\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") "
Jul 10 18:54:43.149828 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.149636    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17def5e7-a73d-4836-a10d-a4b8dd37a860-isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\") pod \"17def5e7-a73d-4836-a10d-a4b8dd37a860\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") "
Jul 10 18:54:43.149828 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.149664    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17def5e7-a73d-4836-a10d-a4b8dd37a860-proxy-tls\") pod \"17def5e7-a73d-4836-a10d-a4b8dd37a860\" (UID: \"17def5e7-a73d-4836-a10d-a4b8dd37a860\") "
Jul 10 18:54:43.149955 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.149915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17def5e7-a73d-4836-a10d-a4b8dd37a860-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17def5e7-a73d-4836-a10d-a4b8dd37a860" (UID: "17def5e7-a73d-4836-a10d-a4b8dd37a860"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:43.150021 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.149995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17def5e7-a73d-4836-a10d-a4b8dd37a860-isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config") pod "17def5e7-a73d-4836-a10d-a4b8dd37a860" (UID: "17def5e7-a73d-4836-a10d-a4b8dd37a860"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:54:43.151796 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.151759    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17def5e7-a73d-4836-a10d-a4b8dd37a860-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17def5e7-a73d-4836-a10d-a4b8dd37a860" (UID: "17def5e7-a73d-4836-a10d-a4b8dd37a860"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:54:43.151900 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.151843    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17def5e7-a73d-4836-a10d-a4b8dd37a860-kube-api-access-w6s28" (OuterVolumeSpecName: "kube-api-access-w6s28") pod "17def5e7-a73d-4836-a10d-a4b8dd37a860" (UID: "17def5e7-a73d-4836-a10d-a4b8dd37a860"). InnerVolumeSpecName "kube-api-access-w6s28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:54:43.250542 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.250439    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17def5e7-a73d-4836-a10d-a4b8dd37a860-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:54:43.250542 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.250473    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6s28\" (UniqueName: \"kubernetes.io/projected/17def5e7-a73d-4836-a10d-a4b8dd37a860-kube-api-access-w6s28\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:54:43.250542 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.250488    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17def5e7-a73d-4836-a10d-a4b8dd37a860-isvc-sklearn-graph-raw-hpa-e9877-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:54:43.250542 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.250504    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17def5e7-a73d-4836-a10d-a4b8dd37a860-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:54:43.337314 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.337277    2561 generic.go:356] "Generic (PLEG): container finished" podID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerID="b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275" exitCode=0
Jul 10 18:54:43.337314 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.337319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerDied","Data":"b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275"}
Jul 10 18:54:43.337522 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.337354    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"
Jul 10 18:54:43.337522 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.337392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb" event={"ID":"17def5e7-a73d-4836-a10d-a4b8dd37a860","Type":"ContainerDied","Data":"43622c0ceb60c5c4f80e51b89b1e236c5b5219f4214f84c92742498bbe8f4214"}
Jul 10 18:54:43.337522 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.337418    2561 scope.go:117] "RemoveContainer" containerID="26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4"
Jul 10 18:54:43.346664 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.346643    2561 scope.go:117] "RemoveContainer" containerID="b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275"
Jul 10 18:54:43.354737 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.354714    2561 scope.go:117] "RemoveContainer" containerID="40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20"
Jul 10 18:54:43.362740 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.362715    2561 scope.go:117] "RemoveContainer" containerID="26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4"
Jul 10 18:54:43.363053 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:54:43.363029    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4\": container with ID starting with 26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4 not found: ID does not exist" containerID="26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4"
Jul 10 18:54:43.363134 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.363060    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4"} err="failed to get container status \"26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4\": rpc error: code = NotFound desc = could not find container \"26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4\": container with ID starting with 26969fe55561c84ea87f1a2b77425adbe8c0c9f95594f08f705c57be7e61a2b4 not found: ID does not exist"
Jul 10 18:54:43.363134 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.363080    2561 scope.go:117] "RemoveContainer" containerID="b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275"
Jul 10 18:54:43.363378 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:54:43.363358    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275\": container with ID starting with b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275 not found: ID does not exist" containerID="b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275"
Jul 10 18:54:43.363449 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.363389    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275"} err="failed to get container status \"b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275\": rpc error: code = NotFound desc = could not find container \"b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275\": container with ID starting with b7d02730c8180e74e6f10a5288ca325e4b44e758f8e70e81dac15e6a3dfa2275 not found: ID does not exist"
Jul 10 18:54:43.363449 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.363412    2561 scope.go:117] "RemoveContainer" containerID="40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20"
Jul 10 18:54:43.363704 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:54:43.363673    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20\": container with ID starting with 40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20 not found: ID does not exist" containerID="40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20"
Jul 10 18:54:43.363762 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.363711    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20"} err="failed to get container status \"40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20\": rpc error: code = NotFound desc = could not find container \"40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20\": container with ID starting with 40d77e2da57b16e87d8ac50d1550c7f5a786b7d70f6938b9e803a5d60cb6fb20 not found: ID does not exist"
Jul 10 18:54:43.364233 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.364213    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"]
Jul 10 18:54:43.371271 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.371242    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e9877-predictor-689f848dd8-jx4fb"]
Jul 10 18:54:43.540773 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:43.540691    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" path="/var/lib/kubelet/pods/17def5e7-a73d-4836-a10d-a4b8dd37a860/volumes"
Jul 10 18:54:45.283423 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:45.283379    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:54:50.284138 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:50.284096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:54:50.284637 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:50.284234    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:54:53.516906 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:53.516871    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:54:53.519310 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:53.519278    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:54:55.283831 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:55.283787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:54:58.522296 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522255    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"]
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522541    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="storage-initializer"
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522552    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="storage-initializer"
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522561    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kube-rbac-proxy"
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522566    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kube-rbac-proxy"
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522578    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container"
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522584    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container"
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522651    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kube-rbac-proxy"
Jul 10 18:54:58.522697 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.522660    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17def5e7-a73d-4836-a10d-a4b8dd37a860" containerName="kserve-container"
Jul 10 18:54:58.527024 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.526998    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.530547 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.530525    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\""
Jul 10 18:54:58.531451 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.531427    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-bb4f7-predictor-serving-cert\""
Jul 10 18:54:58.531521 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.531472    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:54:58.538290 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.538258    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"]
Jul 10 18:54:58.671036 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.670997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2dc777-0b48-4019-9589-ad4bde6e3142-kserve-provision-location\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.671036 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.671035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wh5z\" (UniqueName: \"kubernetes.io/projected/2c2dc777-0b48-4019-9589-ad4bde6e3142-kube-api-access-9wh5z\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.671284 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.671120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.671284 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.671210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2dc777-0b48-4019-9589-ad4bde6e3142-isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.772416 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.772324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2dc777-0b48-4019-9589-ad4bde6e3142-isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.772416 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.772377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2dc777-0b48-4019-9589-ad4bde6e3142-kserve-provision-location\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.772416 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.772405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wh5z\" (UniqueName: \"kubernetes.io/projected/2c2dc777-0b48-4019-9589-ad4bde6e3142-kube-api-access-9wh5z\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.772665 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.772457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.772665 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:54:58.772602    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-serving-cert: secret "isvc-logger-raw-bb4f7-predictor-serving-cert" not found
Jul 10 18:54:58.772734 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:54:58.772679    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls podName:2c2dc777-0b48-4019-9589-ad4bde6e3142 nodeName:}" failed. No retries permitted until 2026-07-10 18:54:59.272657896 +0000 UTC m=+906.226945941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls") pod "isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" (UID: "2c2dc777-0b48-4019-9589-ad4bde6e3142") : secret "isvc-logger-raw-bb4f7-predictor-serving-cert" not found
Jul 10 18:54:58.772898 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.772876    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2dc777-0b48-4019-9589-ad4bde6e3142-kserve-provision-location\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.773062 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.773041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2dc777-0b48-4019-9589-ad4bde6e3142-isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:58.783712 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:58.783680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wh5z\" (UniqueName: \"kubernetes.io/projected/2c2dc777-0b48-4019-9589-ad4bde6e3142-kube-api-access-9wh5z\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:59.276321 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:59.276270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:59.278759 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:59.278718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls\") pod \"isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") " pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:59.438748 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:59.438688    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:54:59.583608 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:54:59.583529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"]
Jul 10 18:54:59.586647 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:54:59.586611    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2dc777_0b48_4019_9589_ad4bde6e3142.slice/crio-b68ad0a8a062c84f6bf3a6c956063e4aaa6602b068c90e23946767687aed1661 WatchSource:0}: Error finding container b68ad0a8a062c84f6bf3a6c956063e4aaa6602b068c90e23946767687aed1661: Status 404 returned error can't find the container with id b68ad0a8a062c84f6bf3a6c956063e4aaa6602b068c90e23946767687aed1661
Jul 10 18:55:00.283705 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:00.283660    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:00.414252 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:00.414205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerStarted","Data":"2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1"}
Jul 10 18:55:00.414422 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:00.414332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerStarted","Data":"b68ad0a8a062c84f6bf3a6c956063e4aaa6602b068c90e23946767687aed1661"}
Jul 10 18:55:04.432965 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:04.432929    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerID="2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1" exitCode=0
Jul 10 18:55:04.433366 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:04.432984    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerDied","Data":"2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1"}
Jul 10 18:55:05.284218 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.284178    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:05.439701 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.439665    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerStarted","Data":"7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71"}
Jul 10 18:55:05.440099 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.439733    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerStarted","Data":"ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c"}
Jul 10 18:55:05.440099 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.439762    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerStarted","Data":"87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16"}
Jul 10 18:55:05.440099 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.440057    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:55:05.440388 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.440121    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:55:05.440388 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.440188    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:55:05.441745 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.441714    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:55:05.442523 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.442503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:05.463235 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:05.463182    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podStartSLOduration=3.627432988 podStartE2EDuration="7.463166198s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83573321s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:55:05.460900942 +0000 UTC m=+912.415189011" watchObservedRunningTime="2026-07-10 18:55:05.463166198 +0000 UTC m=+912.417454259"
Jul 10 18:55:06.444116 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:06.444077    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:55:06.444535 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:06.444510    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:08.461432 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:08.461388    2561 generic.go:356] "Generic (PLEG): container finished" podID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerID="8e3635968d17ade8da50b56b08481b1197a31f28c7719d427d4ad8303127b8f2" exitCode=0
Jul 10 18:55:08.461812 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:08.461462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" event={"ID":"91eea447-c8b2-4279-8ec6-5c0583dba188","Type":"ContainerDied","Data":"8e3635968d17ade8da50b56b08481b1197a31f28c7719d427d4ad8303127b8f2"}
Jul 10 18:55:08.926212 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:08.926186    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:55:09.057394 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.057301    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eea447-c8b2-4279-8ec6-5c0583dba188-proxy-tls\") pod \"91eea447-c8b2-4279-8ec6-5c0583dba188\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") "
Jul 10 18:55:09.057394 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.057366    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91eea447-c8b2-4279-8ec6-5c0583dba188-openshift-service-ca-bundle\") pod \"91eea447-c8b2-4279-8ec6-5c0583dba188\" (UID: \"91eea447-c8b2-4279-8ec6-5c0583dba188\") "
Jul 10 18:55:09.057769 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.057746    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91eea447-c8b2-4279-8ec6-5c0583dba188-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "91eea447-c8b2-4279-8ec6-5c0583dba188" (UID: "91eea447-c8b2-4279-8ec6-5c0583dba188"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:55:09.059495 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.059470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91eea447-c8b2-4279-8ec6-5c0583dba188-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91eea447-c8b2-4279-8ec6-5c0583dba188" (UID: "91eea447-c8b2-4279-8ec6-5c0583dba188"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:09.158091 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.158045    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91eea447-c8b2-4279-8ec6-5c0583dba188-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:55:09.158091 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.158085    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91eea447-c8b2-4279-8ec6-5c0583dba188-openshift-service-ca-bundle\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:55:09.466627 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.466591    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"
Jul 10 18:55:09.466627 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.466609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d" event={"ID":"91eea447-c8b2-4279-8ec6-5c0583dba188","Type":"ContainerDied","Data":"ee7ab470b8b7cbf1532f89152cccbfc2f991bae94c0eaa8de831eaddc74d9931"}
Jul 10 18:55:09.467184 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.466664    2561 scope.go:117] "RemoveContainer" containerID="8e3635968d17ade8da50b56b08481b1197a31f28c7719d427d4ad8303127b8f2"
Jul 10 18:55:09.491950 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.491917    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"]
Jul 10 18:55:09.498124 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.498096    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e9877-646885f67d-vwf6d"]
Jul 10 18:55:09.541111 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:09.541076    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" path="/var/lib/kubelet/pods/91eea447-c8b2-4279-8ec6-5c0583dba188/volumes"
Jul 10 18:55:11.448491 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:11.448461    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:55:11.449092 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:11.449055    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:55:11.449593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:11.449570    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:21.449080 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:21.449028    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:55:21.449574 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:21.449548    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:31.449123 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:31.449075    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:55:31.449613 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:31.449518    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:41.449468 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:41.449418    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:55:41.452033 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:41.449896    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:55:51.449308 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:51.449259    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:55:51.449786 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:55:51.449676    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:56:01.449064 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:01.449014    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:56:01.449562 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:01.449537    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:56:11.449374 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:11.449340    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:56:11.449985 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:11.449869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:56:23.722613 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.722513    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"]
Jul 10 18:56:23.723028 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.722828    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" containerID="cri-o://87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16" gracePeriod=30
Jul 10 18:56:23.723028 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.722916    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy" containerID="cri-o://ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c" gracePeriod=30
Jul 10 18:56:23.724214 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.723951    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" containerID="cri-o://7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71" gracePeriod=30
Jul 10 18:56:23.789294 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.789238    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"]
Jul 10 18:56:23.789544 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.789532    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877"
Jul 10 18:56:23.789587 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.789545    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877"
Jul 10 18:56:23.789621 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.789614    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="91eea447-c8b2-4279-8ec6-5c0583dba188" containerName="model-chainer-raw-hpa-e9877"
Jul 10 18:56:23.796489 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.796340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:23.799992 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.799967    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\""
Jul 10 18:56:23.800164 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.799968    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-f5a60-predictor-serving-cert\""
Jul 10 18:56:23.805667 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.805640    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"]
Jul 10 18:56:23.937618 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.937578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dd825eb-0697-4d5f-a131-5be830283941-isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:23.937803 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.937706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dd825eb-0697-4d5f-a131-5be830283941-proxy-tls\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:23.937803 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.937740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzll4\" (UniqueName: \"kubernetes.io/projected/1dd825eb-0697-4d5f-a131-5be830283941-kube-api-access-kzll4\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:23.937803 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:23.937758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dd825eb-0697-4d5f-a131-5be830283941-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.038989 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.038896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dd825eb-0697-4d5f-a131-5be830283941-proxy-tls\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.038989 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.038939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzll4\" (UniqueName: \"kubernetes.io/projected/1dd825eb-0697-4d5f-a131-5be830283941-kube-api-access-kzll4\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.038989 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.038968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dd825eb-0697-4d5f-a131-5be830283941-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.039279 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.038999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dd825eb-0697-4d5f-a131-5be830283941-isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.039479 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.039457    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dd825eb-0697-4d5f-a131-5be830283941-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.039747 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.039730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dd825eb-0697-4d5f-a131-5be830283941-isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.041357 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.041340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dd825eb-0697-4d5f-a131-5be830283941-proxy-tls\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.048368 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.048343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzll4\" (UniqueName: \"kubernetes.io/projected/1dd825eb-0697-4d5f-a131-5be830283941-kube-api-access-kzll4\") pod \"isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.109504 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.109468    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:24.245517 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.245484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"]
Jul 10 18:56:24.247879 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:56:24.247857    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd825eb_0697_4d5f_a131_5be830283941.slice/crio-ecedf69fca341a950024afdf531e0c6fce7d8e4f9f4ae4a4dd4c8abcc285c4ca WatchSource:0}: Error finding container ecedf69fca341a950024afdf531e0c6fce7d8e4f9f4ae4a4dd4c8abcc285c4ca: Status 404 returned error can't find the container with id ecedf69fca341a950024afdf531e0c6fce7d8e4f9f4ae4a4dd4c8abcc285c4ca
Jul 10 18:56:24.789578 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.789540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerStarted","Data":"39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003"}
Jul 10 18:56:24.790009 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.789603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerStarted","Data":"ecedf69fca341a950024afdf531e0c6fce7d8e4f9f4ae4a4dd4c8abcc285c4ca"}
Jul 10 18:56:24.792205 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.792168    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerID="ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c" exitCode=2
Jul 10 18:56:24.792324 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:24.792236    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerDied","Data":"ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c"}
Jul 10 18:56:26.444629 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:26.444586    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 18:56:28.809867 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:28.809824    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerID="87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16" exitCode=0
Jul 10 18:56:28.810298 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:28.809892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerDied","Data":"87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16"}
Jul 10 18:56:28.811296 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:28.811274    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dd825eb-0697-4d5f-a131-5be830283941" containerID="39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003" exitCode=0
Jul 10 18:56:28.811389 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:28.811317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerDied","Data":"39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003"}
Jul 10 18:56:29.817808 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:29.817777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerStarted","Data":"1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164"}
Jul 10 18:56:29.818250 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:29.817836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerStarted","Data":"baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a"}
Jul 10 18:56:29.818250 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:29.818160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:29.818250 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:29.818210    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:29.819504 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:29.819475    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:56:29.838978 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:29.838923    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podStartSLOduration=2.991621769 podStartE2EDuration="6.838908736s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.847286967s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:56:29.838407472 +0000 UTC m=+996.792695542" watchObservedRunningTime="2026-07-10 18:56:29.838908736 +0000 UTC m=+996.793196802"
Jul 10 18:56:30.821780 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:30.821738    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:56:31.444254 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:31.444213    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 18:56:31.449581 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:31.449546    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:56:31.451205 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:31.451177    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:56:35.826902 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:35.826867    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:56:35.830963 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:35.828710    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:56:36.444494 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:36.444449    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 18:56:36.444672 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:36.444609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:56:41.444519 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:41.444472    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 18:56:41.449936 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:41.449896    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:56:41.450310 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:41.450280    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:56:45.829514 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:45.829469    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:56:46.444440 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:46.444392    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 18:56:51.444315 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:51.444265    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 18:56:51.449783 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:51.449731    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 18:56:51.449955 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:51.449937    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:56:51.450442 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:51.450415    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:56:51.450560 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:51.450545    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:56:53.883386 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.883345    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:56:53.927237 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.927195    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerID="7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71" exitCode=0
Jul 10 18:56:53.927437 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.927269    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"
Jul 10 18:56:53.927437 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.927269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerDied","Data":"7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71"}
Jul 10 18:56:53.927437 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.927341    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt" event={"ID":"2c2dc777-0b48-4019-9589-ad4bde6e3142","Type":"ContainerDied","Data":"b68ad0a8a062c84f6bf3a6c956063e4aaa6602b068c90e23946767687aed1661"}
Jul 10 18:56:53.927437 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.927373    2561 scope.go:117] "RemoveContainer" containerID="7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71"
Jul 10 18:56:53.935738 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.935713    2561 scope.go:117] "RemoveContainer" containerID="ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c"
Jul 10 18:56:53.944083 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.944057    2561 scope.go:117] "RemoveContainer" containerID="87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16"
Jul 10 18:56:53.952104 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.952081    2561 scope.go:117] "RemoveContainer" containerID="2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1"
Jul 10 18:56:53.960124 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.960102    2561 scope.go:117] "RemoveContainer" containerID="7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71"
Jul 10 18:56:53.960472 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:56:53.960452    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71\": container with ID starting with 7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71 not found: ID does not exist" containerID="7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71"
Jul 10 18:56:53.960527 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.960485    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71"} err="failed to get container status \"7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71\": rpc error: code = NotFound desc = could not find container \"7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71\": container with ID starting with 7299953b85a8dabbde2ed01c01802e4dcb60994eb8c0d569883af3953073db71 not found: ID does not exist"
Jul 10 18:56:53.960527 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.960505    2561 scope.go:117] "RemoveContainer" containerID="ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c"
Jul 10 18:56:53.960797 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:56:53.960777    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c\": container with ID starting with ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c not found: ID does not exist" containerID="ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c"
Jul 10 18:56:53.960854 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.960805    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c"} err="failed to get container status \"ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c\": rpc error: code = NotFound desc = could not find container \"ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c\": container with ID starting with ad446744407a5e86a4c5f0b7992da4a6ca856c70625a70b09c0dcc7ff167d88c not found: ID does not exist"
Jul 10 18:56:53.960854 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.960823    2561 scope.go:117] "RemoveContainer" containerID="87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16"
Jul 10 18:56:53.961057 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:56:53.961038    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16\": container with ID starting with 87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16 not found: ID does not exist" containerID="87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16"
Jul 10 18:56:53.961097 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.961064    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16"} err="failed to get container status \"87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16\": rpc error: code = NotFound desc = could not find container \"87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16\": container with ID starting with 87e3ab4d8c690a4723445adb8af12139757cac4d3c93e8b8b7814a6f117a7b16 not found: ID does not exist"
Jul 10 18:56:53.961097 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.961080    2561 scope.go:117] "RemoveContainer" containerID="2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1"
Jul 10 18:56:53.961346 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:56:53.961328    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1\": container with ID starting with 2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1 not found: ID does not exist" containerID="2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1"
Jul 10 18:56:53.961413 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.961348    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1"} err="failed to get container status \"2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1\": rpc error: code = NotFound desc = could not find container \"2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1\": container with ID starting with 2dd7620508e7ab26f791739574f0a5e6de0d75d90c18f33987853430bb13b0a1 not found: ID does not exist"
Jul 10 18:56:53.978773 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.978693    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wh5z\" (UniqueName: \"kubernetes.io/projected/2c2dc777-0b48-4019-9589-ad4bde6e3142-kube-api-access-9wh5z\") pod \"2c2dc777-0b48-4019-9589-ad4bde6e3142\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") "
Jul 10 18:56:53.978773 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.978731    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls\") pod \"2c2dc777-0b48-4019-9589-ad4bde6e3142\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") "
Jul 10 18:56:53.978773 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.978767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2dc777-0b48-4019-9589-ad4bde6e3142-kserve-provision-location\") pod \"2c2dc777-0b48-4019-9589-ad4bde6e3142\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") "
Jul 10 18:56:53.979192 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.979136    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2dc777-0b48-4019-9589-ad4bde6e3142-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c2dc777-0b48-4019-9589-ad4bde6e3142" (UID: "2c2dc777-0b48-4019-9589-ad4bde6e3142"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:56:53.980973 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.980936    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c2dc777-0b48-4019-9589-ad4bde6e3142" (UID: "2c2dc777-0b48-4019-9589-ad4bde6e3142"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:56:53.981094 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:53.980972    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2dc777-0b48-4019-9589-ad4bde6e3142-kube-api-access-9wh5z" (OuterVolumeSpecName: "kube-api-access-9wh5z") pod "2c2dc777-0b48-4019-9589-ad4bde6e3142" (UID: "2c2dc777-0b48-4019-9589-ad4bde6e3142"). InnerVolumeSpecName "kube-api-access-9wh5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:56:54.079347 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.079305    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2dc777-0b48-4019-9589-ad4bde6e3142-isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\") pod \"2c2dc777-0b48-4019-9589-ad4bde6e3142\" (UID: \"2c2dc777-0b48-4019-9589-ad4bde6e3142\") "
Jul 10 18:56:54.079566 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.079549    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wh5z\" (UniqueName: \"kubernetes.io/projected/2c2dc777-0b48-4019-9589-ad4bde6e3142-kube-api-access-9wh5z\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:56:54.079637 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.079601    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2dc777-0b48-4019-9589-ad4bde6e3142-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:56:54.079637 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.079620    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2dc777-0b48-4019-9589-ad4bde6e3142-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:56:54.079709 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.079669    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2dc777-0b48-4019-9589-ad4bde6e3142-isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config") pod "2c2dc777-0b48-4019-9589-ad4bde6e3142" (UID: "2c2dc777-0b48-4019-9589-ad4bde6e3142"). InnerVolumeSpecName "isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:56:54.179926 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.179886    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c2dc777-0b48-4019-9589-ad4bde6e3142-isvc-logger-raw-bb4f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:56:54.250596 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.250556    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"]
Jul 10 18:56:54.253971 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:54.253936    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-bb4f7-predictor-78c88b68dd-5wcbt"]
Jul 10 18:56:55.541259 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:55.541221    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" path="/var/lib/kubelet/pods/2c2dc777-0b48-4019-9589-ad4bde6e3142/volumes"
Jul 10 18:56:55.829255 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:56:55.829134    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:57:05.828756 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:57:05.828714    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:57:15.829261 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:57:15.829214    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:57:25.829685 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:57:25.829635    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:57:35.829325 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:57:35.829277    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:57:45.829430 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:57:45.829332    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:57:55.535947 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:57:55.535895    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:58:05.535938 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:58:05.535887    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:58:15.536340 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:58:15.536289    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:58:25.536051 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:58:25.536006    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:58:35.536062 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:58:35.536004    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:58:45.536902 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:58:45.536855    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:58:55.540599 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:58:55.540569    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:59:04.163256 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163210    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"]
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163555    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163567    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163580    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163606    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="storage-initializer"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163614    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="storage-initializer"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163623    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163630    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163674    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kserve-container"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163682    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="agent"
Jul 10 18:59:04.163708 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.163691    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2dc777-0b48-4019-9589-ad4bde6e3142" containerName="kube-rbac-proxy"
Jul 10 18:59:04.166916 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.166890    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.170026 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.169994    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-37e3de-predictor-serving-cert\""
Jul 10 18:59:04.170528 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.170478    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-37e3de-kube-rbac-proxy-sar-config\""
Jul 10 18:59:04.171605 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.171577    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"]
Jul 10 18:59:04.171879 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.171852    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" containerID="cri-o://baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a" gracePeriod=30
Jul 10 18:59:04.171996 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.171931    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kube-rbac-proxy" containerID="cri-o://1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164" gracePeriod=30
Jul 10 18:59:04.179751 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.179719    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"]
Jul 10 18:59:04.308374 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.308332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634a823b-50a1-443f-862b-e5afaa378932-isvc-primary-37e3de-kube-rbac-proxy-sar-config\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.308374 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.308388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634a823b-50a1-443f-862b-e5afaa378932-proxy-tls\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.308593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.308451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634a823b-50a1-443f-862b-e5afaa378932-kserve-provision-location\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.308593 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.308471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvcw\" (UniqueName: \"kubernetes.io/projected/634a823b-50a1-443f-862b-e5afaa378932-kube-api-access-xjvcw\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.409197 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.409120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634a823b-50a1-443f-862b-e5afaa378932-proxy-tls\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.409421 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.409221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634a823b-50a1-443f-862b-e5afaa378932-kserve-provision-location\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.409421 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.409244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvcw\" (UniqueName: \"kubernetes.io/projected/634a823b-50a1-443f-862b-e5afaa378932-kube-api-access-xjvcw\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.409421 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.409305    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-primary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634a823b-50a1-443f-862b-e5afaa378932-isvc-primary-37e3de-kube-rbac-proxy-sar-config\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.409701 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.409681    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634a823b-50a1-443f-862b-e5afaa378932-kserve-provision-location\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.409968 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.409948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-primary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634a823b-50a1-443f-862b-e5afaa378932-isvc-primary-37e3de-kube-rbac-proxy-sar-config\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.411756 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.411730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634a823b-50a1-443f-862b-e5afaa378932-proxy-tls\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.419194 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.419101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvcw\" (UniqueName: \"kubernetes.io/projected/634a823b-50a1-443f-862b-e5afaa378932-kube-api-access-xjvcw\") pod \"isvc-primary-37e3de-predictor-8557c96d5-d7527\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") " pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.475010 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.474977    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dd825eb-0697-4d5f-a131-5be830283941" containerID="1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164" exitCode=2
Jul 10 18:59:04.475180 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.475017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerDied","Data":"1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164"}
Jul 10 18:59:04.483245 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.483211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:04.631543 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:04.631508    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"]
Jul 10 18:59:04.634483 ip-10-0-134-57 kubenswrapper[2561]: W0710 18:59:04.634454    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634a823b_50a1_443f_862b_e5afaa378932.slice/crio-34807e5173dcbc6cf012815bca6a7b782a2e56a7e3690110d218d50200bc2df7 WatchSource:0}: Error finding container 34807e5173dcbc6cf012815bca6a7b782a2e56a7e3690110d218d50200bc2df7: Status 404 returned error can't find the container with id 34807e5173dcbc6cf012815bca6a7b782a2e56a7e3690110d218d50200bc2df7
Jul 10 18:59:05.481877 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:05.481836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerStarted","Data":"5b74e4be488d303c9d0d612a0d5e687ea3c720656d31f49585551d3e8a5b01f0"}
Jul 10 18:59:05.482306 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:05.481910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerStarted","Data":"34807e5173dcbc6cf012815bca6a7b782a2e56a7e3690110d218d50200bc2df7"}
Jul 10 18:59:05.536799 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:05.536748    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 18:59:05.822131 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:05.822013    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 10 18:59:09.500518 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:09.500477    2561 generic.go:356] "Generic (PLEG): container finished" podID="634a823b-50a1-443f-862b-e5afaa378932" containerID="5b74e4be488d303c9d0d612a0d5e687ea3c720656d31f49585551d3e8a5b01f0" exitCode=0
Jul 10 18:59:09.500983 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:09.500553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerDied","Data":"5b74e4be488d303c9d0d612a0d5e687ea3c720656d31f49585551d3e8a5b01f0"}
Jul 10 18:59:10.507938 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:10.507902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerStarted","Data":"44f39a30ced0326bd60ccb8e45186f3558ad595d03200b62b8d3baebabc58549"}
Jul 10 18:59:10.508398 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:10.507963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerStarted","Data":"ec7c2b53c6df7ac72fef09bdbbeab026748fa58ffe239d8dcb8906974d0c4e44"}
Jul 10 18:59:10.508398 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:10.508284    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:10.508398 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:10.508329    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:10.510054 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:10.510018    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 18:59:10.533187 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:10.533112    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podStartSLOduration=2.6688869520000003 podStartE2EDuration="6.533092763s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.864205811s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:10.531711516 +0000 UTC m=+1157.485999583" watchObservedRunningTime="2026-07-10 18:59:10.533092763 +0000 UTC m=+1157.487380829"
Jul 10 18:59:10.822848 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:10.822741    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 10 18:59:11.513408 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:11.513363    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 18:59:13.833502 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.833471    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:59:13.992185 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.992130    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzll4\" (UniqueName: \"kubernetes.io/projected/1dd825eb-0697-4d5f-a131-5be830283941-kube-api-access-kzll4\") pod \"1dd825eb-0697-4d5f-a131-5be830283941\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") "
Jul 10 18:59:13.992372 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.992223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dd825eb-0697-4d5f-a131-5be830283941-proxy-tls\") pod \"1dd825eb-0697-4d5f-a131-5be830283941\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") "
Jul 10 18:59:13.992372 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.992248    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dd825eb-0697-4d5f-a131-5be830283941-isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\") pod \"1dd825eb-0697-4d5f-a131-5be830283941\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") "
Jul 10 18:59:13.992487 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.992428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dd825eb-0697-4d5f-a131-5be830283941-kserve-provision-location\") pod \"1dd825eb-0697-4d5f-a131-5be830283941\" (UID: \"1dd825eb-0697-4d5f-a131-5be830283941\") "
Jul 10 18:59:13.992703 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.992604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd825eb-0697-4d5f-a131-5be830283941-isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config") pod "1dd825eb-0697-4d5f-a131-5be830283941" (UID: "1dd825eb-0697-4d5f-a131-5be830283941"). InnerVolumeSpecName "isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:59:13.992772 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.992699    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd825eb-0697-4d5f-a131-5be830283941-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dd825eb-0697-4d5f-a131-5be830283941" (UID: "1dd825eb-0697-4d5f-a131-5be830283941"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:59:13.994399 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.994369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd825eb-0697-4d5f-a131-5be830283941-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1dd825eb-0697-4d5f-a131-5be830283941" (UID: "1dd825eb-0697-4d5f-a131-5be830283941"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:59:13.994518 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:13.994457    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd825eb-0697-4d5f-a131-5be830283941-kube-api-access-kzll4" (OuterVolumeSpecName: "kube-api-access-kzll4") pod "1dd825eb-0697-4d5f-a131-5be830283941" (UID: "1dd825eb-0697-4d5f-a131-5be830283941"). InnerVolumeSpecName "kube-api-access-kzll4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:59:14.093917 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.093868    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dd825eb-0697-4d5f-a131-5be830283941-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:59:14.093917 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.093904    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzll4\" (UniqueName: \"kubernetes.io/projected/1dd825eb-0697-4d5f-a131-5be830283941-kube-api-access-kzll4\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:59:14.093917 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.093915    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1dd825eb-0697-4d5f-a131-5be830283941-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:59:14.093917 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.093926    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1dd825eb-0697-4d5f-a131-5be830283941-isvc-sklearn-scale-raw-f5a60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 18:59:14.529089 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.529049    2561 generic.go:356] "Generic (PLEG): container finished" podID="1dd825eb-0697-4d5f-a131-5be830283941" containerID="baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a" exitCode=0
Jul 10 18:59:14.529264 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.529120    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"
Jul 10 18:59:14.529264 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.529125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerDied","Data":"baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a"}
Jul 10 18:59:14.529264 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.529232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5" event={"ID":"1dd825eb-0697-4d5f-a131-5be830283941","Type":"ContainerDied","Data":"ecedf69fca341a950024afdf531e0c6fce7d8e4f9f4ae4a4dd4c8abcc285c4ca"}
Jul 10 18:59:14.529264 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.529260    2561 scope.go:117] "RemoveContainer" containerID="1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164"
Jul 10 18:59:14.537459 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.537439    2561 scope.go:117] "RemoveContainer" containerID="baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a"
Jul 10 18:59:14.545498 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.545477    2561 scope.go:117] "RemoveContainer" containerID="39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003"
Jul 10 18:59:14.553520 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.553501    2561 scope.go:117] "RemoveContainer" containerID="1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164"
Jul 10 18:59:14.553839 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:59:14.553810    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164\": container with ID starting with 1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164 not found: ID does not exist" containerID="1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164"
Jul 10 18:59:14.553894 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.553848    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164"} err="failed to get container status \"1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164\": rpc error: code = NotFound desc = could not find container \"1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164\": container with ID starting with 1e5ad35f5bf3ed4f0062b0fe8eb6ca8f0d106282c418643cd91792b6631e8164 not found: ID does not exist"
Jul 10 18:59:14.553894 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.553870    2561 scope.go:117] "RemoveContainer" containerID="baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a"
Jul 10 18:59:14.554133 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:59:14.554115    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a\": container with ID starting with baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a not found: ID does not exist" containerID="baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a"
Jul 10 18:59:14.554226 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.554163    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a"} err="failed to get container status \"baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a\": rpc error: code = NotFound desc = could not find container \"baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a\": container with ID starting with baee6d39aac2fad23d50db976eede88efc3064ad6ce276ae44fefc6a13c3fb0a not found: ID does not exist"
Jul 10 18:59:14.554226 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.554187    2561 scope.go:117] "RemoveContainer" containerID="39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003"
Jul 10 18:59:14.554432 ip-10-0-134-57 kubenswrapper[2561]: E0710 18:59:14.554411    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003\": container with ID starting with 39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003 not found: ID does not exist" containerID="39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003"
Jul 10 18:59:14.554494 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.554436    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003"} err="failed to get container status \"39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003\": rpc error: code = NotFound desc = could not find container \"39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003\": container with ID starting with 39b3f27c9958facf3ff1ed25cbe8308bb416d1a59ef230ac2369d8b1cae7a003 not found: ID does not exist"
Jul 10 18:59:14.556882 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.556858    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"]
Jul 10 18:59:14.562378 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:14.562352    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-f5a60-predictor-596cc79478-qj8m5"]
Jul 10 18:59:15.540565 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:15.540489    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd825eb-0697-4d5f-a131-5be830283941" path="/var/lib/kubelet/pods/1dd825eb-0697-4d5f-a131-5be830283941/volumes"
Jul 10 18:59:16.519630 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:16.519602    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 18:59:16.520276 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:16.520244    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 18:59:26.520214 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:26.520168    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 18:59:36.520911 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:36.520860    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 18:59:46.520430 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:46.520387    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 18:59:53.545477 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:53.545446    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:59:53.548009 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:53.547984    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 18:59:56.520595 ip-10-0-134-57 kubenswrapper[2561]: I0710 18:59:56.520556    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:00:06.521243 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:06.521193    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:00:16.520753 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:16.520714    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 19:00:24.228613 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228572    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"]
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228882    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="storage-initializer"
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228897    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="storage-initializer"
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228912    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container"
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228918    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container"
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228925    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kube-rbac-proxy"
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228930    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kube-rbac-proxy"
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.228990    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kserve-container"
Jul 10 19:00:24.229057 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.229000    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd825eb-0697-4d5f-a131-5be830283941" containerName="kube-rbac-proxy"
Jul 10 19:00:24.232072 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.232054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.236567 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.236533    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-37e3de-predictor-serving-cert\""
Jul 10 19:00:24.237187 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.237167    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:00:24.237327 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.237223    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-37e3de\""
Jul 10 19:00:24.237444 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.237339    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-37e3de-kube-rbac-proxy-sar-config\""
Jul 10 19:00:24.237907 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.237891    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-37e3de-dockercfg-gxr7r\""
Jul 10 19:00:24.247557 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.247524    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"]
Jul 10 19:00:24.350856 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.350809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04800502-8c3f-4563-bd50-52f40971c65d-kserve-provision-location\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.350856 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.350858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbvvw\" (UniqueName: \"kubernetes.io/projected/04800502-8c3f-4563-bd50-52f40971c65d-kube-api-access-jbvvw\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.351074 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.350893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.351074 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.350994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-cabundle-cert\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.351074 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.351048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-isvc-secondary-37e3de-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.451895 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.451850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-cabundle-cert\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.452100 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.451917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-secondary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-isvc-secondary-37e3de-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.452100 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.451955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04800502-8c3f-4563-bd50-52f40971c65d-kserve-provision-location\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.452100 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.451988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbvvw\" (UniqueName: \"kubernetes.io/projected/04800502-8c3f-4563-bd50-52f40971c65d-kube-api-access-jbvvw\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.452100 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.452027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.452374 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:24.452192    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-serving-cert: secret "isvc-secondary-37e3de-predictor-serving-cert" not found
Jul 10 19:00:24.452374 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:24.452267    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls podName:04800502-8c3f-4563-bd50-52f40971c65d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:24.952246469 +0000 UTC m=+1231.906534528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls") pod "isvc-secondary-37e3de-predictor-696b97fb74-lh44x" (UID: "04800502-8c3f-4563-bd50-52f40971c65d") : secret "isvc-secondary-37e3de-predictor-serving-cert" not found
Jul 10 19:00:24.452447 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.452373    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04800502-8c3f-4563-bd50-52f40971c65d-kserve-provision-location\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.452651 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.452632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-cabundle-cert\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.452687 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.452671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-secondary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-isvc-secondary-37e3de-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.464309 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.464278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbvvw\" (UniqueName: \"kubernetes.io/projected/04800502-8c3f-4563-bd50-52f40971c65d-kube-api-access-jbvvw\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.958430 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.958387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:24.961043 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:24.961009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls\") pod \"isvc-secondary-37e3de-predictor-696b97fb74-lh44x\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") " pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:25.144325 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:25.144285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:25.293046 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:25.293019    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"]
Jul 10 19:00:25.295582 ip-10-0-134-57 kubenswrapper[2561]: W0710 19:00:25.295547    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04800502_8c3f_4563_bd50_52f40971c65d.slice/crio-f5e1bc8b55115cd83043c7ddc45c0ba5f19fdbde4100aafcc92ac17041632292 WatchSource:0}: Error finding container f5e1bc8b55115cd83043c7ddc45c0ba5f19fdbde4100aafcc92ac17041632292: Status 404 returned error can't find the container with id f5e1bc8b55115cd83043c7ddc45c0ba5f19fdbde4100aafcc92ac17041632292
Jul 10 19:00:25.841048 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:25.841008    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x" event={"ID":"04800502-8c3f-4563-bd50-52f40971c65d","Type":"ContainerStarted","Data":"5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713"}
Jul 10 19:00:25.841237 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:25.841080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x" event={"ID":"04800502-8c3f-4563-bd50-52f40971c65d","Type":"ContainerStarted","Data":"f5e1bc8b55115cd83043c7ddc45c0ba5f19fdbde4100aafcc92ac17041632292"}
Jul 10 19:00:27.851291 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:27.851214    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_04800502-8c3f-4563-bd50-52f40971c65d/storage-initializer/0.log"
Jul 10 19:00:27.851291 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:27.851255    2561 generic.go:356] "Generic (PLEG): container finished" podID="04800502-8c3f-4563-bd50-52f40971c65d" containerID="5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713" exitCode=1
Jul 10 19:00:27.851681 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:27.851337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x" event={"ID":"04800502-8c3f-4563-bd50-52f40971c65d","Type":"ContainerDied","Data":"5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713"}
Jul 10 19:00:28.857628 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:28.857595    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_04800502-8c3f-4563-bd50-52f40971c65d/storage-initializer/0.log"
Jul 10 19:00:28.858023 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:28.857680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x" event={"ID":"04800502-8c3f-4563-bd50-52f40971c65d","Type":"ContainerStarted","Data":"f92b11858a1c115a54786ae4c5b90bd119453c9ea226833eb60db56274e9425f"}
Jul 10 19:00:32.878138 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:32.878048    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_04800502-8c3f-4563-bd50-52f40971c65d/storage-initializer/1.log"
Jul 10 19:00:32.878542 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:32.878477    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_04800502-8c3f-4563-bd50-52f40971c65d/storage-initializer/0.log"
Jul 10 19:00:32.878542 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:32.878508    2561 generic.go:356] "Generic (PLEG): container finished" podID="04800502-8c3f-4563-bd50-52f40971c65d" containerID="f92b11858a1c115a54786ae4c5b90bd119453c9ea226833eb60db56274e9425f" exitCode=1
Jul 10 19:00:32.878616 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:32.878587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x" event={"ID":"04800502-8c3f-4563-bd50-52f40971c65d","Type":"ContainerDied","Data":"f92b11858a1c115a54786ae4c5b90bd119453c9ea226833eb60db56274e9425f"}
Jul 10 19:00:32.878666 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:32.878655    2561 scope.go:117] "RemoveContainer" containerID="5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713"
Jul 10 19:00:32.879001 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:32.878984    2561 scope.go:117] "RemoveContainer" containerID="5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713"
Jul 10 19:00:32.889557 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:32.889527    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-37e3de-predictor-696b97fb74-lh44x_kserve-ci-e2e-test(04800502-8c3f-4563-bd50-52f40971c65d)\"" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x" podUID="04800502-8c3f-4563-bd50-52f40971c65d"
Jul 10 19:00:32.889670 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:32.889641    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_kserve-ci-e2e-test_04800502-8c3f-4563-bd50-52f40971c65d_0 in pod sandbox f5e1bc8b55115cd83043c7ddc45c0ba5f19fdbde4100aafcc92ac17041632292 from index: no such id: '5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713'" containerID="5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713"
Jul 10 19:00:32.889719 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:32.889684    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_kserve-ci-e2e-test_04800502-8c3f-4563-bd50-52f40971c65d_0 in pod sandbox f5e1bc8b55115cd83043c7ddc45c0ba5f19fdbde4100aafcc92ac17041632292 from index: no such id: '5097c1aedb1aeac81c2fc5c5aa463333b578ec97200a3d5e844296b8e6247713'"
Jul 10 19:00:33.884887 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:33.884853    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_04800502-8c3f-4563-bd50-52f40971c65d/storage-initializer/1.log"
Jul 10 19:00:38.326646 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.326604    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"]
Jul 10 19:00:38.327108 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.326831    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container" containerID="cri-o://ec7c2b53c6df7ac72fef09bdbbeab026748fa58ffe239d8dcb8906974d0c4e44" gracePeriod=30
Jul 10 19:00:38.327108 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.326894    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kube-rbac-proxy" containerID="cri-o://44f39a30ced0326bd60ccb8e45186f3558ad595d03200b62b8d3baebabc58549" gracePeriod=30
Jul 10 19:00:38.392228 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.392186    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"]
Jul 10 19:00:38.500173 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.500123    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"]
Jul 10 19:00:38.503561 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.503539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.506320 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.506294    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-40cfe0-predictor-serving-cert\""
Jul 10 19:00:38.506863 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.506842    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-40cfe0\""
Jul 10 19:00:38.506984 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.506937    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-40cfe0-dockercfg-vb6gx\""
Jul 10 19:00:38.506984 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.506949    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\""
Jul 10 19:00:38.517847 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.517815    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"]
Jul 10 19:00:38.544048 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.544019    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_04800502-8c3f-4563-bd50-52f40971c65d/storage-initializer/1.log"
Jul 10 19:00:38.546500 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.546475    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:38.663380 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663341    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-isvc-secondary-37e3de-kube-rbac-proxy-sar-config\") pod \"04800502-8c3f-4563-bd50-52f40971c65d\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") "
Jul 10 19:00:38.663580 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663402    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbvvw\" (UniqueName: \"kubernetes.io/projected/04800502-8c3f-4563-bd50-52f40971c65d-kube-api-access-jbvvw\") pod \"04800502-8c3f-4563-bd50-52f40971c65d\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") "
Jul 10 19:00:38.663580 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-cabundle-cert\") pod \"04800502-8c3f-4563-bd50-52f40971c65d\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") "
Jul 10 19:00:38.663580 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663483    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04800502-8c3f-4563-bd50-52f40971c65d-kserve-provision-location\") pod \"04800502-8c3f-4563-bd50-52f40971c65d\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") "
Jul 10 19:00:38.663580 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663540    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls\") pod \"04800502-8c3f-4563-bd50-52f40971c65d\" (UID: \"04800502-8c3f-4563-bd50-52f40971c65d\") "
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6bk\" (UniqueName: \"kubernetes.io/projected/868ea921-2bf6-4c66-a488-80511d4c1b8a-kube-api-access-6h6bk\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-cabundle-cert\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04800502-8c3f-4563-bd50-52f40971c65d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04800502-8c3f-4563-bd50-52f40971c65d" (UID: "04800502-8c3f-4563-bd50-52f40971c65d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868ea921-2bf6-4c66-a488-80511d4c1b8a-kserve-provision-location\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663841    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-isvc-secondary-37e3de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-37e3de-kube-rbac-proxy-sar-config") pod "04800502-8c3f-4563-bd50-52f40971c65d" (UID: "04800502-8c3f-4563-bd50-52f40971c65d"). InnerVolumeSpecName "isvc-secondary-37e3de-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:38.663868 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663849    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04800502-8c3f-4563-bd50-52f40971c65d" (UID: "04800502-8c3f-4563-bd50-52f40971c65d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:38.664215 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.663983    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-isvc-secondary-37e3de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:38.664215 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.664001    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04800502-8c3f-4563-bd50-52f40971c65d-cabundle-cert\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:38.664215 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.664016    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04800502-8c3f-4563-bd50-52f40971c65d-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:38.665691 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.665659    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04800502-8c3f-4563-bd50-52f40971c65d-kube-api-access-jbvvw" (OuterVolumeSpecName: "kube-api-access-jbvvw") pod "04800502-8c3f-4563-bd50-52f40971c65d" (UID: "04800502-8c3f-4563-bd50-52f40971c65d"). InnerVolumeSpecName "kube-api-access-jbvvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:38.665806 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.665723    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04800502-8c3f-4563-bd50-52f40971c65d" (UID: "04800502-8c3f-4563-bd50-52f40971c65d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:38.765077 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.765307 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765091    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.765307 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6bk\" (UniqueName: \"kubernetes.io/projected/868ea921-2bf6-4c66-a488-80511d4c1b8a-kube-api-access-6h6bk\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.765307 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-cabundle-cert\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.765307 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868ea921-2bf6-4c66-a488-80511d4c1b8a-kserve-provision-location\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.765307 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:38.765262    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-serving-cert: secret "isvc-init-fail-40cfe0-predictor-serving-cert" not found
Jul 10 19:00:38.765500 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765331    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04800502-8c3f-4563-bd50-52f40971c65d-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:38.765500 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:38.765344    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls podName:868ea921-2bf6-4c66-a488-80511d4c1b8a nodeName:}" failed. No retries permitted until 2026-07-10 19:00:39.265321452 +0000 UTC m=+1246.219609501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls") pod "isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" (UID: "868ea921-2bf6-4c66-a488-80511d4c1b8a") : secret "isvc-init-fail-40cfe0-predictor-serving-cert" not found
Jul 10 19:00:38.765500 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765368    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbvvw\" (UniqueName: \"kubernetes.io/projected/04800502-8c3f-4563-bd50-52f40971c65d-kube-api-access-jbvvw\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:38.765661 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868ea921-2bf6-4c66-a488-80511d4c1b8a-kserve-provision-location\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.765918 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.765957 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.765903    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-cabundle-cert\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.775986 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.775950    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6bk\" (UniqueName: \"kubernetes.io/projected/868ea921-2bf6-4c66-a488-80511d4c1b8a-kube-api-access-6h6bk\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:38.906843 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.906809    2561 generic.go:356] "Generic (PLEG): container finished" podID="634a823b-50a1-443f-862b-e5afaa378932" containerID="44f39a30ced0326bd60ccb8e45186f3558ad595d03200b62b8d3baebabc58549" exitCode=2
Jul 10 19:00:38.907018 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.906881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerDied","Data":"44f39a30ced0326bd60ccb8e45186f3558ad595d03200b62b8d3baebabc58549"}
Jul 10 19:00:38.908199 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.908172    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-37e3de-predictor-696b97fb74-lh44x_04800502-8c3f-4563-bd50-52f40971c65d/storage-initializer/1.log"
Jul 10 19:00:38.908286 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.908228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x" event={"ID":"04800502-8c3f-4563-bd50-52f40971c65d","Type":"ContainerDied","Data":"f5e1bc8b55115cd83043c7ddc45c0ba5f19fdbde4100aafcc92ac17041632292"}
Jul 10 19:00:38.908286 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.908271    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"
Jul 10 19:00:38.908432 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.908289    2561 scope.go:117] "RemoveContainer" containerID="f92b11858a1c115a54786ae4c5b90bd119453c9ea226833eb60db56274e9425f"
Jul 10 19:00:38.961343 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.961306    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"]
Jul 10 19:00:38.966277 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:38.966244    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-37e3de-predictor-696b97fb74-lh44x"]
Jul 10 19:00:39.269599 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:39.269501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:39.271967 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:39.271944    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls\") pod \"isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") " pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:39.417113 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:39.417066    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:39.541702 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:39.541625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04800502-8c3f-4563-bd50-52f40971c65d" path="/var/lib/kubelet/pods/04800502-8c3f-4563-bd50-52f40971c65d/volumes"
Jul 10 19:00:39.553213 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:39.553170    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"]
Jul 10 19:00:39.555636 ip-10-0-134-57 kubenswrapper[2561]: W0710 19:00:39.555604    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868ea921_2bf6_4c66_a488_80511d4c1b8a.slice/crio-d5435775421e96c4c56458fc99c97e8a7cc60db237d1ed845212077f9db8a613 WatchSource:0}: Error finding container d5435775421e96c4c56458fc99c97e8a7cc60db237d1ed845212077f9db8a613: Status 404 returned error can't find the container with id d5435775421e96c4c56458fc99c97e8a7cc60db237d1ed845212077f9db8a613
Jul 10 19:00:39.915732 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:39.915687    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" event={"ID":"868ea921-2bf6-4c66-a488-80511d4c1b8a","Type":"ContainerStarted","Data":"43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd"}
Jul 10 19:00:39.916004 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:39.915760    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" event={"ID":"868ea921-2bf6-4c66-a488-80511d4c1b8a","Type":"ContainerStarted","Data":"d5435775421e96c4c56458fc99c97e8a7cc60db237d1ed845212077f9db8a613"}
Jul 10 19:00:41.514275 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:41.514095    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 10 19:00:42.931734 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.931693    2561 generic.go:356] "Generic (PLEG): container finished" podID="634a823b-50a1-443f-862b-e5afaa378932" containerID="ec7c2b53c6df7ac72fef09bdbbeab026748fa58ffe239d8dcb8906974d0c4e44" exitCode=0
Jul 10 19:00:42.932089 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.931755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerDied","Data":"ec7c2b53c6df7ac72fef09bdbbeab026748fa58ffe239d8dcb8906974d0c4e44"}
Jul 10 19:00:42.983539 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.983513    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 19:00:42.998727 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.998693    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjvcw\" (UniqueName: \"kubernetes.io/projected/634a823b-50a1-443f-862b-e5afaa378932-kube-api-access-xjvcw\") pod \"634a823b-50a1-443f-862b-e5afaa378932\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") "
Jul 10 19:00:42.998870 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.998753    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634a823b-50a1-443f-862b-e5afaa378932-kserve-provision-location\") pod \"634a823b-50a1-443f-862b-e5afaa378932\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") "
Jul 10 19:00:42.998870 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.998786    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634a823b-50a1-443f-862b-e5afaa378932-proxy-tls\") pod \"634a823b-50a1-443f-862b-e5afaa378932\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") "
Jul 10 19:00:42.998870 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.998835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-primary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634a823b-50a1-443f-862b-e5afaa378932-isvc-primary-37e3de-kube-rbac-proxy-sar-config\") pod \"634a823b-50a1-443f-862b-e5afaa378932\" (UID: \"634a823b-50a1-443f-862b-e5afaa378932\") "
Jul 10 19:00:42.999102 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.999074    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634a823b-50a1-443f-862b-e5afaa378932-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "634a823b-50a1-443f-862b-e5afaa378932" (UID: "634a823b-50a1-443f-862b-e5afaa378932"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:42.999298 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:42.999273    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634a823b-50a1-443f-862b-e5afaa378932-isvc-primary-37e3de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-37e3de-kube-rbac-proxy-sar-config") pod "634a823b-50a1-443f-862b-e5afaa378932" (UID: "634a823b-50a1-443f-862b-e5afaa378932"). InnerVolumeSpecName "isvc-primary-37e3de-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:43.001011 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.000983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634a823b-50a1-443f-862b-e5afaa378932-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "634a823b-50a1-443f-862b-e5afaa378932" (UID: "634a823b-50a1-443f-862b-e5afaa378932"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:43.001110 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.000987    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634a823b-50a1-443f-862b-e5afaa378932-kube-api-access-xjvcw" (OuterVolumeSpecName: "kube-api-access-xjvcw") pod "634a823b-50a1-443f-862b-e5afaa378932" (UID: "634a823b-50a1-443f-862b-e5afaa378932"). InnerVolumeSpecName "kube-api-access-xjvcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:43.100339 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.100234    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-37e3de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634a823b-50a1-443f-862b-e5afaa378932-isvc-primary-37e3de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:43.100339 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.100273    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvcw\" (UniqueName: \"kubernetes.io/projected/634a823b-50a1-443f-862b-e5afaa378932-kube-api-access-xjvcw\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:43.100339 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.100287    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634a823b-50a1-443f-862b-e5afaa378932-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:43.100339 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.100328    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634a823b-50a1-443f-862b-e5afaa378932-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:43.937607 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.937579    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"
Jul 10 19:00:43.938067 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.937571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527" event={"ID":"634a823b-50a1-443f-862b-e5afaa378932","Type":"ContainerDied","Data":"34807e5173dcbc6cf012815bca6a7b782a2e56a7e3690110d218d50200bc2df7"}
Jul 10 19:00:43.938067 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.937711    2561 scope.go:117] "RemoveContainer" containerID="44f39a30ced0326bd60ccb8e45186f3558ad595d03200b62b8d3baebabc58549"
Jul 10 19:00:43.939515 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.939498    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_868ea921-2bf6-4c66-a488-80511d4c1b8a/storage-initializer/0.log"
Jul 10 19:00:43.939626 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.939529    2561 generic.go:356] "Generic (PLEG): container finished" podID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerID="43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd" exitCode=1
Jul 10 19:00:43.939626 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.939588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" event={"ID":"868ea921-2bf6-4c66-a488-80511d4c1b8a","Type":"ContainerDied","Data":"43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd"}
Jul 10 19:00:43.946775 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.946751    2561 scope.go:117] "RemoveContainer" containerID="ec7c2b53c6df7ac72fef09bdbbeab026748fa58ffe239d8dcb8906974d0c4e44"
Jul 10 19:00:43.958468 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.958444    2561 scope.go:117] "RemoveContainer" containerID="5b74e4be488d303c9d0d612a0d5e687ea3c720656d31f49585551d3e8a5b01f0"
Jul 10 19:00:43.989873 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.989846    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"]
Jul 10 19:00:43.995012 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:43.994973    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-37e3de-predictor-8557c96d5-d7527"]
Jul 10 19:00:44.946251 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:44.946221    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_868ea921-2bf6-4c66-a488-80511d4c1b8a/storage-initializer/0.log"
Jul 10 19:00:44.946636 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:44.946319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" event={"ID":"868ea921-2bf6-4c66-a488-80511d4c1b8a","Type":"ContainerStarted","Data":"3105613531db2319e705274113b77d7d8b2e169ecc1bf201efbe17e47064c68f"}
Jul 10 19:00:45.541700 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:45.541660    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634a823b-50a1-443f-862b-e5afaa378932" path="/var/lib/kubelet/pods/634a823b-50a1-443f-862b-e5afaa378932/volumes"
Jul 10 19:00:47.960532 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:47.960502    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_868ea921-2bf6-4c66-a488-80511d4c1b8a/storage-initializer/1.log"
Jul 10 19:00:47.960961 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:47.960914    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_868ea921-2bf6-4c66-a488-80511d4c1b8a/storage-initializer/0.log"
Jul 10 19:00:47.960961 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:47.960945    2561 generic.go:356] "Generic (PLEG): container finished" podID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerID="3105613531db2319e705274113b77d7d8b2e169ecc1bf201efbe17e47064c68f" exitCode=1
Jul 10 19:00:47.961043 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:47.961021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" event={"ID":"868ea921-2bf6-4c66-a488-80511d4c1b8a","Type":"ContainerDied","Data":"3105613531db2319e705274113b77d7d8b2e169ecc1bf201efbe17e47064c68f"}
Jul 10 19:00:47.961100 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:47.961089    2561 scope.go:117] "RemoveContainer" containerID="43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd"
Jul 10 19:00:47.961487 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:47.961470    2561 scope.go:117] "RemoveContainer" containerID="43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd"
Jul 10 19:00:47.972009 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:47.971982    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_kserve-ci-e2e-test(868ea921-2bf6-4c66-a488-80511d4c1b8a)\"" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a"
Jul 10 19:00:47.972158 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:00:47.972112    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_kserve-ci-e2e-test_868ea921-2bf6-4c66-a488-80511d4c1b8a_0 in pod sandbox d5435775421e96c4c56458fc99c97e8a7cc60db237d1ed845212077f9db8a613 from index: no such id: '43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd'" containerID="43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd"
Jul 10 19:00:47.972214 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:47.972188    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_kserve-ci-e2e-test_868ea921-2bf6-4c66-a488-80511d4c1b8a_0 in pod sandbox d5435775421e96c4c56458fc99c97e8a7cc60db237d1ed845212077f9db8a613 from index: no such id: '43618ae5d8d154cb2ffba110193cbb283cbc03fab414538ad01b0b6ed95404dd'"
Jul 10 19:00:48.446874 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.446838    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"]
Jul 10 19:00:48.572442 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572408    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"]
Jul 10 19:00:48.572722 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572710    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container"
Jul 10 19:00:48.572766 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572724    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container"
Jul 10 19:00:48.572766 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572738    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kube-rbac-proxy"
Jul 10 19:00:48.572766 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572743    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kube-rbac-proxy"
Jul 10 19:00:48.572766 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572751    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04800502-8c3f-4563-bd50-52f40971c65d" containerName="storage-initializer"
Jul 10 19:00:48.572766 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572756    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04800502-8c3f-4563-bd50-52f40971c65d" containerName="storage-initializer"
Jul 10 19:00:48.572766 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572766    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="storage-initializer"
Jul 10 19:00:48.572949 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572771    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="storage-initializer"
Jul 10 19:00:48.572949 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572850    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kserve-container"
Jul 10 19:00:48.572949 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572858    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04800502-8c3f-4563-bd50-52f40971c65d" containerName="storage-initializer"
Jul 10 19:00:48.572949 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572867    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="634a823b-50a1-443f-862b-e5afaa378932" containerName="kube-rbac-proxy"
Jul 10 19:00:48.572949 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572913    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04800502-8c3f-4563-bd50-52f40971c65d" containerName="storage-initializer"
Jul 10 19:00:48.572949 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572918    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="04800502-8c3f-4563-bd50-52f40971c65d" containerName="storage-initializer"
Jul 10 19:00:48.573117 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.572968    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="04800502-8c3f-4563-bd50-52f40971c65d" containerName="storage-initializer"
Jul 10 19:00:48.577282 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.577251    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.581204 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.581175    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\""
Jul 10 19:00:48.581372 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.581238    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-3fcc6-predictor-serving-cert\""
Jul 10 19:00:48.581372 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.581359    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-kr64x\""
Jul 10 19:00:48.593049 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.593018    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"]
Jul 10 19:00:48.642183 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.642128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96f75d2-dc23-4cdc-baca-809892dedc44-kserve-provision-location\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.642367 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.642210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b96f75d2-dc23-4cdc-baca-809892dedc44-proxy-tls\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.642367 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.642284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b96f75d2-dc23-4cdc-baca-809892dedc44-raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.642367 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.642331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7bq\" (UniqueName: \"kubernetes.io/projected/b96f75d2-dc23-4cdc-baca-809892dedc44-kube-api-access-sq7bq\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.742797 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.742692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7bq\" (UniqueName: \"kubernetes.io/projected/b96f75d2-dc23-4cdc-baca-809892dedc44-kube-api-access-sq7bq\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.742797 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.742750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96f75d2-dc23-4cdc-baca-809892dedc44-kserve-provision-location\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.742797 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.742790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b96f75d2-dc23-4cdc-baca-809892dedc44-proxy-tls\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.743054 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.742872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b96f75d2-dc23-4cdc-baca-809892dedc44-raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.743356 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.743335    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96f75d2-dc23-4cdc-baca-809892dedc44-kserve-provision-location\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.743662 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.743640    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b96f75d2-dc23-4cdc-baca-809892dedc44-raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.745577 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.745552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b96f75d2-dc23-4cdc-baca-809892dedc44-proxy-tls\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.752208 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.752176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7bq\" (UniqueName: \"kubernetes.io/projected/b96f75d2-dc23-4cdc-baca-809892dedc44-kube-api-access-sq7bq\") pod \"raw-sklearn-3fcc6-predictor-6b698db54-qskc9\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") " pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.888444 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.888403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:48.967911 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:48.967887    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_868ea921-2bf6-4c66-a488-80511d4c1b8a/storage-initializer/1.log"
Jul 10 19:00:49.035783 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.035756    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"]
Jul 10 19:00:49.041355 ip-10-0-134-57 kubenswrapper[2561]: W0710 19:00:49.041323    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96f75d2_dc23_4cdc_baca_809892dedc44.slice/crio-70b3555d074880881d2c94ed83308899e04e65d81a12aabb3f14617ff0eb9c5e WatchSource:0}: Error finding container 70b3555d074880881d2c94ed83308899e04e65d81a12aabb3f14617ff0eb9c5e: Status 404 returned error can't find the container with id 70b3555d074880881d2c94ed83308899e04e65d81a12aabb3f14617ff0eb9c5e
Jul 10 19:00:49.109644 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.109618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_868ea921-2bf6-4c66-a488-80511d4c1b8a/storage-initializer/1.log"
Jul 10 19:00:49.113214 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.113188    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:49.151219 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.146096    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868ea921-2bf6-4c66-a488-80511d4c1b8a-kserve-provision-location\") pod \"868ea921-2bf6-4c66-a488-80511d4c1b8a\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") "
Jul 10 19:00:49.151219 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.146211    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h6bk\" (UniqueName: \"kubernetes.io/projected/868ea921-2bf6-4c66-a488-80511d4c1b8a-kube-api-access-6h6bk\") pod \"868ea921-2bf6-4c66-a488-80511d4c1b8a\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") "
Jul 10 19:00:49.151219 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.146336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls\") pod \"868ea921-2bf6-4c66-a488-80511d4c1b8a\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") "
Jul 10 19:00:49.151219 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.146367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-cabundle-cert\") pod \"868ea921-2bf6-4c66-a488-80511d4c1b8a\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") "
Jul 10 19:00:49.151219 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.146436    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\") pod \"868ea921-2bf6-4c66-a488-80511d4c1b8a\" (UID: \"868ea921-2bf6-4c66-a488-80511d4c1b8a\") "
Jul 10 19:00:49.151599 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.151398    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config") pod "868ea921-2bf6-4c66-a488-80511d4c1b8a" (UID: "868ea921-2bf6-4c66-a488-80511d4c1b8a"). InnerVolumeSpecName "isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:49.151599 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.151479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868ea921-2bf6-4c66-a488-80511d4c1b8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "868ea921-2bf6-4c66-a488-80511d4c1b8a" (UID: "868ea921-2bf6-4c66-a488-80511d4c1b8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:49.151680 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.151605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "868ea921-2bf6-4c66-a488-80511d4c1b8a" (UID: "868ea921-2bf6-4c66-a488-80511d4c1b8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:49.153442 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.153390    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "868ea921-2bf6-4c66-a488-80511d4c1b8a" (UID: "868ea921-2bf6-4c66-a488-80511d4c1b8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:49.153582 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.153544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868ea921-2bf6-4c66-a488-80511d4c1b8a-kube-api-access-6h6bk" (OuterVolumeSpecName: "kube-api-access-6h6bk") pod "868ea921-2bf6-4c66-a488-80511d4c1b8a" (UID: "868ea921-2bf6-4c66-a488-80511d4c1b8a"). InnerVolumeSpecName "kube-api-access-6h6bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:49.248111 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.248063    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-isvc-init-fail-40cfe0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:49.248111 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.248115    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/868ea921-2bf6-4c66-a488-80511d4c1b8a-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:49.248336 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.248132    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h6bk\" (UniqueName: \"kubernetes.io/projected/868ea921-2bf6-4c66-a488-80511d4c1b8a-kube-api-access-6h6bk\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:49.248336 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.248164    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/868ea921-2bf6-4c66-a488-80511d4c1b8a-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:49.248336 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.248185    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/868ea921-2bf6-4c66-a488-80511d4c1b8a-cabundle-cert\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:00:49.973488 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.973458    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7_868ea921-2bf6-4c66-a488-80511d4c1b8a/storage-initializer/1.log"
Jul 10 19:00:49.973926 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.973568    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"
Jul 10 19:00:49.973926 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.973574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7" event={"ID":"868ea921-2bf6-4c66-a488-80511d4c1b8a","Type":"ContainerDied","Data":"d5435775421e96c4c56458fc99c97e8a7cc60db237d1ed845212077f9db8a613"}
Jul 10 19:00:49.973926 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.973650    2561 scope.go:117] "RemoveContainer" containerID="3105613531db2319e705274113b77d7d8b2e169ecc1bf201efbe17e47064c68f"
Jul 10 19:00:49.975668 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.975636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerStarted","Data":"4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f"}
Jul 10 19:00:49.975774 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:49.975745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerStarted","Data":"70b3555d074880881d2c94ed83308899e04e65d81a12aabb3f14617ff0eb9c5e"}
Jul 10 19:00:50.032488 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:50.032448    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"]
Jul 10 19:00:50.036497 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:50.036464    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-40cfe0-predictor-5b45d7457b-xfdx7"]
Jul 10 19:00:51.540521 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:51.540489    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a" path="/var/lib/kubelet/pods/868ea921-2bf6-4c66-a488-80511d4c1b8a/volumes"
Jul 10 19:00:52.990885 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:52.990852    2561 generic.go:356] "Generic (PLEG): container finished" podID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerID="4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f" exitCode=0
Jul 10 19:00:52.991265 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:52.990934    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerDied","Data":"4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f"}
Jul 10 19:00:53.996624 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:53.996584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerStarted","Data":"9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb"}
Jul 10 19:00:53.997016 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:53.996650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerStarted","Data":"e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750"}
Jul 10 19:00:53.997016 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:53.996853    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:54.020685 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:54.020623    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podStartSLOduration=2.310391998 podStartE2EDuration="6.020602288s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.71021029s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:54.019009817 +0000 UTC m=+1260.973297914" watchObservedRunningTime="2026-07-10 19:00:54.020602288 +0000 UTC m=+1260.974890356"
Jul 10 19:00:55.001349 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:55.001319    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:00:55.002521 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:55.002497    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:00:56.005366 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:00:56.005327    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:01:01.010553 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:01:01.010517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:01:01.012332 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:01:01.012289    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:01:11.012112 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:01:11.012066    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:01:21.011908 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:01:21.011860    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:01:31.011519 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:01:31.011471    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:01:41.011098 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:01:41.011051    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:01:51.011179 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:01:51.011116    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:02:01.012319 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:01.012287    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:02:08.836051 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.836014    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"]
Jul 10 19:02:08.836513 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.836323    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerName="storage-initializer"
Jul 10 19:02:08.836513 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.836335    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerName="storage-initializer"
Jul 10 19:02:08.836513 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.836355    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerName="storage-initializer"
Jul 10 19:02:08.836513 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.836362    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerName="storage-initializer"
Jul 10 19:02:08.836513 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.836412    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerName="storage-initializer"
Jul 10 19:02:08.836513 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.836419    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="868ea921-2bf6-4c66-a488-80511d4c1b8a" containerName="storage-initializer"
Jul 10 19:02:08.839598 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.839575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:08.842872 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.842848    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-6e4de-predictor-serving-cert\""
Jul 10 19:02:08.843107 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.843090    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\""
Jul 10 19:02:08.859596 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.859561    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"]
Jul 10 19:02:08.900312 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.900273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f910ba-aeaf-4585-b88a-dbd66d88bb64-raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:08.900523 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.900328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkwkt\" (UniqueName: \"kubernetes.io/projected/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kube-api-access-xkwkt\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:08.900523 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.900437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f910ba-aeaf-4585-b88a-dbd66d88bb64-proxy-tls\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:08.900523 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.900480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kserve-provision-location\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:08.943736 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.943694    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"]
Jul 10 19:02:08.944019 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.943989    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" containerID="cri-o://e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750" gracePeriod=30
Jul 10 19:02:08.944129 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:08.944024    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kube-rbac-proxy" containerID="cri-o://9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb" gracePeriod=30
Jul 10 19:02:09.001496 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.001463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f910ba-aeaf-4585-b88a-dbd66d88bb64-proxy-tls\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.001496 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.001502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kserve-provision-location\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.001759 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.001529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f910ba-aeaf-4585-b88a-dbd66d88bb64-raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.001759 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.001574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkwkt\" (UniqueName: \"kubernetes.io/projected/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kube-api-access-xkwkt\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.001982 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.001957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kserve-provision-location\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.002343 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.002322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f910ba-aeaf-4585-b88a-dbd66d88bb64-raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.004022 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.004002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f910ba-aeaf-4585-b88a-dbd66d88bb64-proxy-tls\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.012847 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.012813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkwkt\" (UniqueName: \"kubernetes.io/projected/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kube-api-access-xkwkt\") pod \"raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.149931 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.149888    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:09.303426 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.303398    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"]
Jul 10 19:02:09.305864 ip-10-0-134-57 kubenswrapper[2561]: W0710 19:02:09.305828    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f910ba_aeaf_4585_b88a_dbd66d88bb64.slice/crio-acaee0ab1af46423fa781fff0267f1c17c99140917d2fb8bf1a0f70965eae9b3 WatchSource:0}: Error finding container acaee0ab1af46423fa781fff0267f1c17c99140917d2fb8bf1a0f70965eae9b3: Status 404 returned error can't find the container with id acaee0ab1af46423fa781fff0267f1c17c99140917d2fb8bf1a0f70965eae9b3
Jul 10 19:02:09.317634 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.317603    2561 generic.go:356] "Generic (PLEG): container finished" podID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerID="9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb" exitCode=2
Jul 10 19:02:09.317767 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.317672    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerDied","Data":"9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb"}
Jul 10 19:02:09.319076 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:09.319049    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerStarted","Data":"acaee0ab1af46423fa781fff0267f1c17c99140917d2fb8bf1a0f70965eae9b3"}
Jul 10 19:02:10.324887 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:10.324846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerStarted","Data":"0815c4158d5ba5eaba1046f3a529bf5d6055860de3aa652fce4bceaf77203a9d"}
Jul 10 19:02:11.005984 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:11.005940    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 10 19:02:11.011521 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:11.011478    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:02:13.339679 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.339643    2561 generic.go:356] "Generic (PLEG): container finished" podID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerID="0815c4158d5ba5eaba1046f3a529bf5d6055860de3aa652fce4bceaf77203a9d" exitCode=0
Jul 10 19:02:13.340096 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.339714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerDied","Data":"0815c4158d5ba5eaba1046f3a529bf5d6055860de3aa652fce4bceaf77203a9d"}
Jul 10 19:02:13.897350 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.897321    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:02:13.943189 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.943123    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b96f75d2-dc23-4cdc-baca-809892dedc44-raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\") pod \"b96f75d2-dc23-4cdc-baca-809892dedc44\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") "
Jul 10 19:02:13.943189 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.943196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b96f75d2-dc23-4cdc-baca-809892dedc44-proxy-tls\") pod \"b96f75d2-dc23-4cdc-baca-809892dedc44\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") "
Jul 10 19:02:13.943445 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.943229    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96f75d2-dc23-4cdc-baca-809892dedc44-kserve-provision-location\") pod \"b96f75d2-dc23-4cdc-baca-809892dedc44\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") "
Jul 10 19:02:13.943445 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.943264    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq7bq\" (UniqueName: \"kubernetes.io/projected/b96f75d2-dc23-4cdc-baca-809892dedc44-kube-api-access-sq7bq\") pod \"b96f75d2-dc23-4cdc-baca-809892dedc44\" (UID: \"b96f75d2-dc23-4cdc-baca-809892dedc44\") "
Jul 10 19:02:13.943616 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.943586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96f75d2-dc23-4cdc-baca-809892dedc44-raw-sklearn-3fcc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-3fcc6-kube-rbac-proxy-sar-config") pod "b96f75d2-dc23-4cdc-baca-809892dedc44" (UID: "b96f75d2-dc23-4cdc-baca-809892dedc44"). InnerVolumeSpecName "raw-sklearn-3fcc6-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:02:13.943673 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.943598    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96f75d2-dc23-4cdc-baca-809892dedc44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b96f75d2-dc23-4cdc-baca-809892dedc44" (UID: "b96f75d2-dc23-4cdc-baca-809892dedc44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:02:13.945359 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.945331    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96f75d2-dc23-4cdc-baca-809892dedc44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b96f75d2-dc23-4cdc-baca-809892dedc44" (UID: "b96f75d2-dc23-4cdc-baca-809892dedc44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:02:13.945576 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:13.945549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96f75d2-dc23-4cdc-baca-809892dedc44-kube-api-access-sq7bq" (OuterVolumeSpecName: "kube-api-access-sq7bq") pod "b96f75d2-dc23-4cdc-baca-809892dedc44" (UID: "b96f75d2-dc23-4cdc-baca-809892dedc44"). InnerVolumeSpecName "kube-api-access-sq7bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:02:14.044209 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.044070    2561 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b96f75d2-dc23-4cdc-baca-809892dedc44-raw-sklearn-3fcc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:02:14.044209 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.044121    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b96f75d2-dc23-4cdc-baca-809892dedc44-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:02:14.044209 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.044133    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b96f75d2-dc23-4cdc-baca-809892dedc44-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:02:14.044209 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.044164    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq7bq\" (UniqueName: \"kubernetes.io/projected/b96f75d2-dc23-4cdc-baca-809892dedc44-kube-api-access-sq7bq\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:02:14.346367 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.346272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerStarted","Data":"761927f080cf9a26dd277aaf8db56c9c8ee24a7218980a2beaf8ffffa6024790"}
Jul 10 19:02:14.346367 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.346348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerStarted","Data":"9d6bf34ac6b6b4a0485f0fba5435d53d104b7afb50581d590f40e7477ff05555"}
Jul 10 19:02:14.346835 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.346651    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:14.346835 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.346829    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:14.348421 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.348390    2561 generic.go:356] "Generic (PLEG): container finished" podID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerID="e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750" exitCode=0
Jul 10 19:02:14.348572 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.348420    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:02:14.348572 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.348459    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"
Jul 10 19:02:14.348572 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.348482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerDied","Data":"e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750"}
Jul 10 19:02:14.348572 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.348551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9" event={"ID":"b96f75d2-dc23-4cdc-baca-809892dedc44","Type":"ContainerDied","Data":"70b3555d074880881d2c94ed83308899e04e65d81a12aabb3f14617ff0eb9c5e"}
Jul 10 19:02:14.348770 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.348592    2561 scope.go:117] "RemoveContainer" containerID="9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb"
Jul 10 19:02:14.357191 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.357168    2561 scope.go:117] "RemoveContainer" containerID="e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750"
Jul 10 19:02:14.365267 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.365242    2561 scope.go:117] "RemoveContainer" containerID="4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f"
Jul 10 19:02:14.373257 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.373230    2561 scope.go:117] "RemoveContainer" containerID="9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb"
Jul 10 19:02:14.373587 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:02:14.373567    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb\": container with ID starting with 9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb not found: ID does not exist" containerID="9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb"
Jul 10 19:02:14.373672 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.373598    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb"} err="failed to get container status \"9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb\": rpc error: code = NotFound desc = could not find container \"9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb\": container with ID starting with 9b3020cafd8b90edd9fa5415e1886d5bb0f677701ec82500218c93b7df0e02cb not found: ID does not exist"
Jul 10 19:02:14.373672 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.373618    2561 scope.go:117] "RemoveContainer" containerID="e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750"
Jul 10 19:02:14.373891 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:02:14.373876    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750\": container with ID starting with e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750 not found: ID does not exist" containerID="e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750"
Jul 10 19:02:14.373933 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.373895    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750"} err="failed to get container status \"e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750\": rpc error: code = NotFound desc = could not find container \"e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750\": container with ID starting with e02014fafc55938ac5cc8958b5284823b2a10019d0acdb6f32fe6b81e2d54750 not found: ID does not exist"
Jul 10 19:02:14.373933 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.373911    2561 scope.go:117] "RemoveContainer" containerID="4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f"
Jul 10 19:02:14.374177 ip-10-0-134-57 kubenswrapper[2561]: E0710 19:02:14.374137    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f\": container with ID starting with 4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f not found: ID does not exist" containerID="4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f"
Jul 10 19:02:14.374268 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.374180    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f"} err="failed to get container status \"4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f\": rpc error: code = NotFound desc = could not find container \"4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f\": container with ID starting with 4c94296d3a4d02617f51441757ba9475164f7d0cec38fafe33ee6ca176e86a0f not found: ID does not exist"
Jul 10 19:02:14.381477 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.381425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podStartSLOduration=2.905789992 podStartE2EDuration="6.381409826s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.475619834s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:14.380191061 +0000 UTC m=+1341.334479131" watchObservedRunningTime="2026-07-10 19:02:14.381409826 +0000 UTC m=+1341.335697893"
Jul 10 19:02:14.407127 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.407092    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"]
Jul 10 19:02:14.409160 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:14.409122    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-3fcc6-predictor-6b698db54-qskc9"]
Jul 10 19:02:15.353728 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:15.353635    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:02:15.544637 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:15.544598    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" path="/var/lib/kubelet/pods/b96f75d2-dc23-4cdc-baca-809892dedc44/volumes"
Jul 10 19:02:20.359080 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:20.359046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:02:20.359635 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:20.359606    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:02:30.359613 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:30.359563    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:02:40.359585 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:40.359540    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:02:50.359950 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:02:50.359906    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:03:00.359913 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:00.359864    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:03:10.360278 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:10.360230    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:03:20.360338 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:20.360308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:03:28.944166 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:28.944104    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"]
Jul 10 19:03:28.944764 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:28.944694    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" containerID="cri-o://9d6bf34ac6b6b4a0485f0fba5435d53d104b7afb50581d590f40e7477ff05555" gracePeriod=30
Jul 10 19:03:28.945077 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:28.945024    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kube-rbac-proxy" containerID="cri-o://761927f080cf9a26dd277aaf8db56c9c8ee24a7218980a2beaf8ffffa6024790" gracePeriod=30
Jul 10 19:03:29.669046 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:29.669011    2561 generic.go:356] "Generic (PLEG): container finished" podID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerID="761927f080cf9a26dd277aaf8db56c9c8ee24a7218980a2beaf8ffffa6024790" exitCode=2
Jul 10 19:03:29.669241 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:29.669060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerDied","Data":"761927f080cf9a26dd277aaf8db56c9c8ee24a7218980a2beaf8ffffa6024790"}
Jul 10 19:03:30.354172 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:30.354093    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 10 19:03:30.360596 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:30.360563    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:03:33.687964 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.687934    2561 generic.go:356] "Generic (PLEG): container finished" podID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerID="9d6bf34ac6b6b4a0485f0fba5435d53d104b7afb50581d590f40e7477ff05555" exitCode=0
Jul 10 19:03:33.688406 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.688008    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerDied","Data":"9d6bf34ac6b6b4a0485f0fba5435d53d104b7afb50581d590f40e7477ff05555"}
Jul 10 19:03:33.688406 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.688067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8" event={"ID":"79f910ba-aeaf-4585-b88a-dbd66d88bb64","Type":"ContainerDied","Data":"acaee0ab1af46423fa781fff0267f1c17c99140917d2fb8bf1a0f70965eae9b3"}
Jul 10 19:03:33.688406 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.688099    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acaee0ab1af46423fa781fff0267f1c17c99140917d2fb8bf1a0f70965eae9b3"
Jul 10 19:03:33.700680 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.700653    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:03:33.766943 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.766835    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f910ba-aeaf-4585-b88a-dbd66d88bb64-raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\") pod \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") "
Jul 10 19:03:33.766943 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.766881    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f910ba-aeaf-4585-b88a-dbd66d88bb64-proxy-tls\") pod \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") "
Jul 10 19:03:33.767226 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.766970    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkwkt\" (UniqueName: \"kubernetes.io/projected/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kube-api-access-xkwkt\") pod \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") "
Jul 10 19:03:33.767226 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.767033    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kserve-provision-location\") pod \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\" (UID: \"79f910ba-aeaf-4585-b88a-dbd66d88bb64\") "
Jul 10 19:03:33.767330 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.767226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f910ba-aeaf-4585-b88a-dbd66d88bb64-raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config") pod "79f910ba-aeaf-4585-b88a-dbd66d88bb64" (UID: "79f910ba-aeaf-4585-b88a-dbd66d88bb64"). InnerVolumeSpecName "raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:03:33.767447 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.767422    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79f910ba-aeaf-4585-b88a-dbd66d88bb64" (UID: "79f910ba-aeaf-4585-b88a-dbd66d88bb64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:33.769062 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.769040    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f910ba-aeaf-4585-b88a-dbd66d88bb64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79f910ba-aeaf-4585-b88a-dbd66d88bb64" (UID: "79f910ba-aeaf-4585-b88a-dbd66d88bb64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:03:33.769155 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.769075    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kube-api-access-xkwkt" (OuterVolumeSpecName: "kube-api-access-xkwkt") pod "79f910ba-aeaf-4585-b88a-dbd66d88bb64" (UID: "79f910ba-aeaf-4585-b88a-dbd66d88bb64"). InnerVolumeSpecName "kube-api-access-xkwkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:03:33.867671 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.867636    2561 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79f910ba-aeaf-4585-b88a-dbd66d88bb64-raw-sklearn-runtime-6e4de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.867671 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.867665    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79f910ba-aeaf-4585-b88a-dbd66d88bb64-proxy-tls\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.867671 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.867678    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkwkt\" (UniqueName: \"kubernetes.io/projected/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kube-api-access-xkwkt\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:03:33.867972 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:33.867689    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79f910ba-aeaf-4585-b88a-dbd66d88bb64-kserve-provision-location\") on node \"ip-10-0-134-57.ec2.internal\" DevicePath \"\""
Jul 10 19:03:34.692038 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:34.692000    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"
Jul 10 19:03:34.716250 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:34.716209    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"]
Jul 10 19:03:34.720800 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:34.720768    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6e4de-predictor-66cd464d98-klwg8"]
Jul 10 19:03:35.540525 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:03:35.540489    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" path="/var/lib/kubelet/pods/79f910ba-aeaf-4585-b88a-dbd66d88bb64/volumes"
Jul 10 19:04:01.743838 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:01.743741    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-krjlw_3c91b6f5-d69d-47d5-9782-18d0a40d6f4e/global-pull-secret-syncer/0.log"
Jul 10 19:04:01.995795 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:01.995693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zw7sz_dfd4ef1f-f28c-4979-bc40-20da7f09a697/konnectivity-agent/0.log"
Jul 10 19:04:02.079004 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:02.078975    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-57.ec2.internal_fc0950c7cf4a4407bce2f7a337f490f9/haproxy/0.log"
Jul 10 19:04:07.104991 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:07.104956    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tmm6s_76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a/node-exporter/0.log"
Jul 10 19:04:07.140861 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:07.140828    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tmm6s_76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a/kube-rbac-proxy/0.log"
Jul 10 19:04:07.184490 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:07.184459    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tmm6s_76c6b8d8-f57b-4e2a-9a4d-ead51ed0435a/init-textfile/0.log"
Jul 10 19:04:09.610974 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:09.610917    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/1.log"
Jul 10 19:04:09.615484 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:09.615459    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-jc755_a06ad978-e5e9-44ee-9250-b743cdeefd5e/console-operator/2.log"
Jul 10 19:04:10.228853 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.228817    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"]
Jul 10 19:04:10.229113 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229101    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="storage-initializer"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229114    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="storage-initializer"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229125    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229131    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229140    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229160    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229169    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kube-rbac-proxy"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229174    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kube-rbac-proxy"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229181    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="storage-initializer"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229187    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="storage-initializer"
Jul 10 19:04:10.229197 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229201    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kube-rbac-proxy"
Jul 10 19:04:10.229510 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229207    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kube-rbac-proxy"
Jul 10 19:04:10.229510 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229261    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kube-rbac-proxy"
Jul 10 19:04:10.229510 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229273    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96f75d2-dc23-4cdc-baca-809892dedc44" containerName="kserve-container"
Jul 10 19:04:10.229510 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229279    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kserve-container"
Jul 10 19:04:10.229510 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.229286    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f910ba-aeaf-4585-b88a-dbd66d88bb64" containerName="kube-rbac-proxy"
Jul 10 19:04:10.232599 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.232575    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.235580 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.235553    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fs9kw\"/\"openshift-service-ca.crt\""
Jul 10 19:04:10.236710 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.236687    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fs9kw\"/\"default-dockercfg-9qq2d\""
Jul 10 19:04:10.236827 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.236732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fs9kw\"/\"kube-root-ca.crt\""
Jul 10 19:04:10.245442 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.245411    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"]
Jul 10 19:04:10.350982 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.350941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-sys\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.351182 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.350992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-proc\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.351182 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.351018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-podres\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.351182 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.351048    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-lib-modules\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.351182 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.351066    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwgw\" (UniqueName: \"kubernetes.io/projected/663677df-fce3-47fe-8ae3-22877d49dc7d-kube-api-access-2hwgw\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451700 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-sys\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451788 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-proc\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451788 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451756    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-podres\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451788 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-lib-modules\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451989 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-sys\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451989 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwgw\" (UniqueName: \"kubernetes.io/projected/663677df-fce3-47fe-8ae3-22877d49dc7d-kube-api-access-2hwgw\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451989 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-proc\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451989 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-podres\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.451989 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.451961    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/663677df-fce3-47fe-8ae3-22877d49dc7d-lib-modules\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.461073 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.461046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwgw\" (UniqueName: \"kubernetes.io/projected/663677df-fce3-47fe-8ae3-22877d49dc7d-kube-api-access-2hwgw\") pod \"perf-node-gather-daemonset-c6h88\" (UID: \"663677df-fce3-47fe-8ae3-22877d49dc7d\") " pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.542390 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.542284    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.667263 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.667233    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-dn5rj_02fcf8cf-9311-4b87-a8ba-0d29f7a6f147/volume-data-source-validator/0.log"
Jul 10 19:04:10.680936 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.680909    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"]
Jul 10 19:04:10.683718 ip-10-0-134-57 kubenswrapper[2561]: W0710 19:04:10.683692    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod663677df_fce3_47fe_8ae3_22877d49dc7d.slice/crio-c759381185dbc335fa2657c5286e5c5b8165ccdba97b4a0594fd9b6d2c50b1de WatchSource:0}: Error finding container c759381185dbc335fa2657c5286e5c5b8165ccdba97b4a0594fd9b6d2c50b1de: Status 404 returned error can't find the container with id c759381185dbc335fa2657c5286e5c5b8165ccdba97b4a0594fd9b6d2c50b1de
Jul 10 19:04:10.851331 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.851293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88" event={"ID":"663677df-fce3-47fe-8ae3-22877d49dc7d","Type":"ContainerStarted","Data":"3f5875f26007ff8928da78b7b4f4de81349ebed6e4b0d585edef2d3701844697"}
Jul 10 19:04:10.851508 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.851349    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88" event={"ID":"663677df-fce3-47fe-8ae3-22877d49dc7d","Type":"ContainerStarted","Data":"c759381185dbc335fa2657c5286e5c5b8165ccdba97b4a0594fd9b6d2c50b1de"}
Jul 10 19:04:10.851508 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.851383    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:10.872819 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:10.872763    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88" podStartSLOduration=0.872747912 podStartE2EDuration="872.747912ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:10.871556491 +0000 UTC m=+1457.825844560" watchObservedRunningTime="2026-07-10 19:04:10.872747912 +0000 UTC m=+1457.827035979"
Jul 10 19:04:11.716887 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:11.716855    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9d5q_d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1/dns/0.log"
Jul 10 19:04:11.743959 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:11.743926    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9d5q_d6cc27e1-ba64-4b3f-b9dd-edf842e7f9b1/kube-rbac-proxy/0.log"
Jul 10 19:04:11.836112 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:11.836084    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hhb4q_ee32311a-4aa2-4168-bffd-f355e8dae244/dns-node-resolver/0.log"
Jul 10 19:04:12.354653 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:12.354618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8cvcp_3f921aa9-6e25-4f9c-a45c-3eb718f655a9/node-ca/0.log"
Jul 10 19:04:13.676161 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:13.676128    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rdwq2_01dc8b49-c2dd-42af-a45d-9dac0564b514/serve-healthcheck-canary/0.log"
Jul 10 19:04:14.094422 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:14.094335    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-4dz8j_d91b43b9-931d-417d-bc0b-9609ead67393/insights-operator/1.log"
Jul 10 19:04:14.094422 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:14.094411    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-4dz8j_d91b43b9-931d-417d-bc0b-9609ead67393/insights-operator/0.log"
Jul 10 19:04:14.283122 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:14.283092    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t4x7p_29dabca5-17b1-40dd-9a4b-f8f473ef8d52/kube-rbac-proxy/0.log"
Jul 10 19:04:14.314860 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:14.314823    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t4x7p_29dabca5-17b1-40dd-9a4b-f8f473ef8d52/exporter/0.log"
Jul 10 19:04:14.343287 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:14.343257    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t4x7p_29dabca5-17b1-40dd-9a4b-f8f473ef8d52/extractor/0.log"
Jul 10 19:04:16.445215 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:16.445187    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-659578d74d-grz59_3170a30b-51bc-486b-81bb-a40ff980832c/manager/0.log"
Jul 10 19:04:16.679553 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:16.679520    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-nqbd7_4beb8329-e0c3-48a0-b3a9-3ece6de9ae08/seaweedfs/0.log"
Jul 10 19:04:16.866065 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:16.865987    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fs9kw/perf-node-gather-daemonset-c6h88"
Jul 10 19:04:20.903030 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:20.903001    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-n6662_5a4a213a-999c-402e-b83f-440a927acb98/migrator/0.log"
Jul 10 19:04:20.930158 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:20.930109    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-n6662_5a4a213a-999c-402e-b83f-440a927acb98/graceful-termination/0.log"
Jul 10 19:04:21.330121 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:21.330088    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-n8zbc_ac31c4b7-c733-47fd-8c91-8db7c161c319/kube-storage-version-migrator-operator/1.log"
Jul 10 19:04:21.331205 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:21.331186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-n8zbc_ac31c4b7-c733-47fd-8c91-8db7c161c319/kube-storage-version-migrator-operator/0.log"
Jul 10 19:04:22.217250 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:22.217219    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gkfp_e48d716c-b5ee-4a31-b532-b815b3b58626/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:04:22.304321 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:22.304289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gkfp_e48d716c-b5ee-4a31-b532-b815b3b58626/egress-router-binary-copy/0.log"
Jul 10 19:04:22.339786 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:22.339758    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gkfp_e48d716c-b5ee-4a31-b532-b815b3b58626/cni-plugins/0.log"
Jul 10 19:04:22.366627 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:22.366603    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gkfp_e48d716c-b5ee-4a31-b532-b815b3b58626/bond-cni-plugin/0.log"
Jul 10 19:04:22.395277 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:22.395254    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gkfp_e48d716c-b5ee-4a31-b532-b815b3b58626/routeoverride-cni/0.log"
Jul 10 19:04:22.423041 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:22.423006    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gkfp_e48d716c-b5ee-4a31-b532-b815b3b58626/whereabouts-cni-bincopy/0.log"
Jul 10 19:04:22.464530 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:22.464449    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gkfp_e48d716c-b5ee-4a31-b532-b815b3b58626/whereabouts-cni/0.log"
Jul 10 19:04:23.084526 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:23.084493    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kkdrl_0132a4d3-2442-4059-bdb6-28f3b2fbc18d/kube-multus/0.log"
Jul 10 19:04:23.112539 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:23.112447    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4sbhh_f6e8e427-7654-41f4-a0cf-8b4e07fc6381/network-metrics-daemon/0.log"
Jul 10 19:04:23.137140 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:23.137100    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4sbhh_f6e8e427-7654-41f4-a0cf-8b4e07fc6381/kube-rbac-proxy/0.log"
Jul 10 19:04:24.348669 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.348637    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/ovn-controller/0.log"
Jul 10 19:04:24.374475 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.374444    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/ovn-acl-logging/0.log"
Jul 10 19:04:24.405654 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.405621    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/kube-rbac-proxy-node/0.log"
Jul 10 19:04:24.435192 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.435162    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/kube-rbac-proxy-ovn-metrics/0.log"
Jul 10 19:04:24.459972 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.459943    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/northd/0.log"
Jul 10 19:04:24.490778 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.490754    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/nbdb/0.log"
Jul 10 19:04:24.521780 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.521751    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/sbdb/0.log"
Jul 10 19:04:24.579819 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:24.579783    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nltht_cc3dac8e-46b2-4462-9b80-f1ddd3732eed/ovnkube-controller/0.log"
Jul 10 19:04:26.078816 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:26.078785    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fjdnw_6b52971f-2c3f-4986-a372-2c8d98ea58da/network-check-target-container/0.log"
Jul 10 19:04:27.109525 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:27.109495    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-f267f_d968d33f-746c-4f85-8118-64bf2e41a3ee/iptables-alerter/0.log"
Jul 10 19:04:27.998242 ip-10-0-134-57 kubenswrapper[2561]: I0710 19:04:27.998207    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-z58qt_13fb85dc-3804-4c03-9735-fe08cd156f0d/tuned/0.log"