Jul 02 13:14:52.700038 ip-10-0-133-122 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 13:14:53.055755 ip-10-0-133-122 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:53.055755 ip-10-0-133-122 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 13:14:53.055755 ip-10-0-133-122 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:53.055755 ip-10-0-133-122 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 13:14:53.055755 ip-10-0-133-122 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 13:14:53.056689 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.056596    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 13:14:53.060978 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.060957    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.060978 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.060978    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.060984    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.060987    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.060990    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.060994    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.060998    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061001    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061005    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061007    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061010    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061019    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061022    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061024    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061027    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061032    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061036    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061039    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061041    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061044    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.061052 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061047    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061050    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061052    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061055    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061058    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061061    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061064    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061066    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061069    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061072    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061074    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061077    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061083    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061086    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061088    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061090    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061093    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061096    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061098    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.061495 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061100    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061103    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061107    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061109    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061112    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061114    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061117    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061119    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061121    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061124    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061126    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061129    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061132    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061135    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061138    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061141    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061144    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061147    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061149    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.061967 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061152    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061154    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061158    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061160    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061163    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061165    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061168    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061173    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061176    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061178    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061181    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061184    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061186    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061189    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061191    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061194    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061196    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061198    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061203    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.062442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061207    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061209    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061212    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061215    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061217    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061220    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061222    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061225    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061227    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061231    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061234    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061236    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061239    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061241    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061244    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061247    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061249    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061252    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061254    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061257    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.062900 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061259    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.061262    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062020    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062029    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062032    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062035    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062039    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062041    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062044    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062046    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062049    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062051    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062054    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062056    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062059    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062062    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062064    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062068    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062071    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062074    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.063362 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062077    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062080    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062084    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062088    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062091    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062094    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062096    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062099    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062102    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062104    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062108    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062110    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062112    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062115    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062126    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062128    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062132    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062134    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062137    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.063897 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062142    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062145    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062149    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062152    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062155    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062162    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062164    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062167    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062170    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062173    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062176    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062178    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062181    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062184    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062188    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062191    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062193    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062196    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062198    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.064401 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062201    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062204    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062206    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062209    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062211    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062214    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062217    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062219    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062222    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062225    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062228    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062230    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062233    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062235    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062238    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062240    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062243    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062245    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062248    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062250    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.064875 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062252    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062255    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062258    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062261    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062264    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062267    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062270    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062273    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062275    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062278    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062280    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062283    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062285    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062288    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062290    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062293    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062295    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062298    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062300    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062302    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.065372 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062305    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062307    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062311    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062685    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062690    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062693    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062695    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062698    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062717    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062721    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062723    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062726    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062729    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062732    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062735    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062738    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062740    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062743    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062746    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062748    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.065867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062751    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062754    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062757    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062759    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062763    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062767    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062770    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062772    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062775    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062778    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062780    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062783    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062785    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062788    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062791    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062794    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062797    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062799    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062802    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.066344 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062804    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062807    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062810    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062812    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062815    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062817    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062820    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062823    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062825    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062828    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062830    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062833    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062835    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062838    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062840    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062842    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062845    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062847    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062850    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062852    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.066810 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062855    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062859    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062861    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062864    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062866    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062868    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062871    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062873    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062876    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062879    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062882    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062886    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062889    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062892    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062895    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062897    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062900    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062903    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062905    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062909    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.067285 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062911    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062914    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062917    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062919    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062923    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062927    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062930    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062933    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062936    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062938    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062941    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062944    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062950    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062952    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062955    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062958    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062960    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062963    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062966    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.067777 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062968    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062970    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062974    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.062977    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063059    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063068    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063085    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063090    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063094    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063097    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063102    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063106    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063111    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063114    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063118    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063121    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063124    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063127    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063130    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063133    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063136    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063139    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063143    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063146    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 13:14:53.068238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063149    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063154    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063158    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063166    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063169    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063173    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063176    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063179    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063182    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063185    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063188    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063192    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063196    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063200    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063203    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063206    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063209    2555 flags.go:64] FLAG: --enable-server="true"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063212    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063217    2555 flags.go:64] FLAG: --event-burst="100"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063220    2555 flags.go:64] FLAG: --event-qps="50"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063222    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063225    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063228    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063232    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 13:14:53.068840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063236    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063238    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063241    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063244    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063247    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063250    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063252    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063255    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063258    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063263    2555 flags.go:64] FLAG: --feature-gates=""
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063267    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063270    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063273    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063276    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063279    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063282    2555 flags.go:64] FLAG: --help="false"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063285    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063288    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063291    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063294    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063298    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063301    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063304    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 13:14:53.069406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063307    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063309    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063312    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063315    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063318    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063321    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063324    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063327    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063329    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063332    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063335    2555 flags.go:64] FLAG: --lock-file=""
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063338    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063341    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063344    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063350    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063352    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063355    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063358    2555 flags.go:64] FLAG: --logging-format="text"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063362    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063366    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063369    2555 flags.go:64] FLAG: --manifest-url=""
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063372    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063376    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063379    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063383    2555 flags.go:64] FLAG: --max-pods="110"
Jul 02 13:14:53.070106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063386    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063389    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063392    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063395    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063398    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063402    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063405    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063413    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063416    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063419    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063422    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063425    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063430    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063433    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063436    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063439    2555 flags.go:64] FLAG: --port="10250"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063442    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063444    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03507a5e71c69341b"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063447    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063450    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063453    2555 flags.go:64] FLAG: --register-node="true"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063456    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063459    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063462    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 02 13:14:53.070718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063465    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063468    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063472    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063475    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063478    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063481    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063483    2555 flags.go:64] FLAG: --runonce="false"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063486    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063489    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063492    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063494    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063498    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063501    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063504    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063508    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063510    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063514    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063516    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063519    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063522    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063524    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063527    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063532    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063535    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063537    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 13:14:53.071534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063541    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063544    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063547    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063550    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063553    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063555    2555 flags.go:64] FLAG: --v="2"
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063560    2555 flags.go:64] FLAG: --version="false"
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063564    2555 flags.go:64] FLAG: --vmodule=""
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063569    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063572    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063687    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063690    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063694    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063697    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063712    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063715    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063718    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063721    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063726    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063728    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063731    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063733    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.072397 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063737    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063740    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063742    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063745    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063748    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063750    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063753    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063755    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063758    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063761    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063764    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063767    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063769    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063772    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063774    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063777    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063780    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063782    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063786    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063789    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.072961 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063792    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063794    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063796    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063800    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063803    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063805    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063808    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063810    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063816    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063818    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063821    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063826    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063829    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063833    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063836    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063839    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063842    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063845    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063848    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.073442 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063850    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063853    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063856    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063858    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063861    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063864    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063867    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063869    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063872    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063874    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063877    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063882    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063884    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063887    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063889    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063892    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063894    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063897    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063900    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.074005 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063902    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063905    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063910    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063912    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063915    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063918    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063920    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063923    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063926    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063929    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063931    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063934    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063936    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063939    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063941    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063944    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063946    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063950    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063954    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063957    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.074466 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063959    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063962    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063966    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063969    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063973    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063976    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063979    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063982    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.063984    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.063990    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.073077    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.073095    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.073804    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.073812    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 13:14:53.074965 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073874    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073879    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073882    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073885    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073888    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073891    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073894    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073897    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073900    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073902    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073906    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073910    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073914    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073917    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073920    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073923    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073926    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073930    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073934    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.075320 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073937    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073940    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073943    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073945    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073948    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073951    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073954    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073957    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073960    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073963    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073966    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073970    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073972    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073975    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073977    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073980    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073983    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073986    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073988    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073991    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.075805 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073993    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073996    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.073999    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074001    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074004    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074007    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074009    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074012    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074015    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074017    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074020    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074023    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074026    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074028    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074031    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074033    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074036    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074039    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074041    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074044    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.076298 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074047    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074050    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074053    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074056    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074058    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074061    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074063    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074066    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074069    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074071    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074074    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074076    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074078    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074081    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074083    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074086    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074089    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074091    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074094    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.076795 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074098    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074102    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074105    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074108    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074112    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074114    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074117    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074120    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074122    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074125    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074128    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074130    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074133    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074136    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074139    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074142    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074145    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074147    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074150    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074153    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.077289 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074156    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.074161    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074286    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074292    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074295    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074298    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074301    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074304    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074306    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074309    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074311    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074314    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074316    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074319    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 13:14:53.077867 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074322    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074325    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074327    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074330    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074332    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074335    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074337    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074340    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074343    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074345    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074348    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074350    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074353    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074356    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074359    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074362    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074364    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074367    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074370    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074372    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 13:14:53.078214 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074375    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074377    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074380    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074383    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074385    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074388    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074390    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074393    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074395    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074398    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074400    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074403    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074405    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074408    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074410    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074413    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074416    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074418    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074420    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074423    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 13:14:53.078684 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074426    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074429    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074432    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074434    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074437    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074440    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074443    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074445    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074448    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074450    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074453    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074455    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074457    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074460    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074463    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074467    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074470    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074473    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074476    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 13:14:53.079184 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074480    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074482    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074485    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074488    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074490    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074493    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074495    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074499    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074502    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074504    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074507    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074509    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074512    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074514    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074517    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074520    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074524    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074527    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074530    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074533    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 13:14:53.079633 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074535    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074538    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074540    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074543    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074546    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074548    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074551    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:53.074553    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.074558    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.074677    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.076873    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.078189    2555 server.go:1019] "Starting client certificate rotation"
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.078289    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:53.080123 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.078327    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 13:14:53.080461 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.080318    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:53.081631 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.081608    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 13:14:53.093114 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.093092    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 02 13:14:53.095567 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.095548    2555 log.go:25] "Validated CRI v1 image API"
Jul 02 13:14:53.097083 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.097065    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 13:14:53.101332 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.101307    2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b38890c-5961-41d7-bc24-08f019de3169:/dev/nvme0n1p3 8efbfa6c-5af1-4205-80d9-da52539003e2:/dev/nvme0n1p4]
Jul 02 13:14:53.101430 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.101337    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 13:14:53.107247 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.107077    2555 manager.go:216] Machine: {Timestamp:2026-07-02 13:14:53.105852139 +0000 UTC m=+0.296181800 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099520 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2134d4d67804f24c2bf77c95525c2a SystemUUID:ec2134d4-d678-04f2-4c2b-f77c95525c2a BootID:f8a802e8-3281-43bb-a33d-44cc4f40d45b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:03:00:63:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:03:00:63:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:35:a7:44:52:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 13:14:53.107247 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.107229    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 13:14:53.107430 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.107322    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 13:14:53.109444 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.109426    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:53.110371 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.110337    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 13:14:53.110576 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.110393    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 13:14:53.111217 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.111205    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 13:14:53.111269 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.111224    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 13:14:53.111269 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.111235    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:53.111937 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.111924    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 13:14:53.113001 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.112989    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 13:14:53.118100 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118078    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:53.118253 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118241    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 13:14:53.118309 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118284    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 13:14:53.118309 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118299    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 13:14:53.118408 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118315    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 13:14:53.118408 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118330    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 02 13:14:53.118408 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118343    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 13:14:53.118836 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118821    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:53.118903 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.118846    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 13:14:53.121065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.121043    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 13:14:53.122633 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.122614    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 13:14:53.122746 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.122650    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 13:14:53.123693 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123682    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 13:14:53.123756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123698    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 13:14:53.123756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123720    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 13:14:53.123756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123726    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 13:14:53.123756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123736    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 13:14:53.123756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123741    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 13:14:53.123756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123749    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 13:14:53.123756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123755    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 13:14:53.123929 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123762    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 13:14:53.123929 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123768    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 13:14:53.123929 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123776    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 13:14:53.123929 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.123782    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 13:14:53.124503 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.124492    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 13:14:53.124503 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.124503    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 13:14:53.128021 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.127999    2555 server.go:1268] "Started kubelet"
Jul 02 13:14:53.128097 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.128038    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 13:14:53.128147 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.128100    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 13:14:53.128312 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.128187    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 13:14:53.128423 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.128406    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 13:14:53.128540 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.128490    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 13:14:53.128738 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.128696    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 13:14:53.129129 ip-10-0-133-122 systemd[1]: Started Kubernetes Kubelet.
Jul 02 13:14:53.129977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.129864    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 13:14:53.130336 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.130287    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 13:14:53.133224 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.133204    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p6zpv"
Jul 02 13:14:53.134411 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.134381    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 13:14:53.135208 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.134398    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-122.ec2.internal.18be7b51a8177ffd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-122.ec2.internal,UID:ip-10-0-133-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-122.ec2.internal,},FirstTimestamp:2026-07-02 13:14:53.127974909 +0000 UTC m=+0.318304572,LastTimestamp:2026-07-02 13:14:53.127974909 +0000 UTC m=+0.318304572,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-122.ec2.internal,}"
Jul 02 13:14:53.137783 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.137764    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:53.137864 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.137773    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 13:14:53.137977 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.137956    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.138423 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.138404    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 13:14:53.138423 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.138421    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 13:14:53.138540 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.138494    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 13:14:53.138582 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.138571    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 13:14:53.138582 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.138578    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 13:14:53.139861 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.139841    2555 factory.go:55] Registering systemd factory
Jul 02 13:14:53.139957 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.139877    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 02 13:14:53.140136 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.140119    2555 factory.go:153] Registering CRI-O factory
Jul 02 13:14:53.140216 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.140142    2555 factory.go:223] Registration of the crio container factory successfully
Jul 02 13:14:53.140317 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.140303    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 13:14:53.140370 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.140341    2555 factory.go:103] Registering Raw factory
Jul 02 13:14:53.140370 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.140363    2555 manager.go:1196] Started watching for new ooms in manager
Jul 02 13:14:53.140508 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.140412    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p6zpv"
Jul 02 13:14:53.140862 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.140837    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 13:14:53.140932 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.140907    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 13:14:53.140932 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.140920    2555 manager.go:349] Starting recovery of all containers
Jul 02 13:14:53.153176 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.153154    2555 manager.go:354] Recovery completed
Jul 02 13:14:53.157605 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.157591    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:53.159530 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.159511    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:53.159605 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.159544    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:53.159605 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.159556    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:53.160057 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.160042    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 13:14:53.160057 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.160054    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 13:14:53.160057 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.160058    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:53.160211 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.160070    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 13:14:53.160745 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.160728    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:53.160796 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.160759    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:53.160796 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.160769    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:53.162043 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.162031    2555 policy_none.go:49] "None policy: Start"
Jul 02 13:14:53.162077 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.162048    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 13:14:53.162077 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.162058    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 13:14:53.163899 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.163888    2555 policy_none.go:47] "Start"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.209397    2555 manager.go:337] "Starting Device Plugin manager"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.209438    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.209451    2555 server.go:87] "Starting device plugin registration server"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.209741    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.209754    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.209878    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.210013    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.210023    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.210692    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.210763    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.218332    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.219445    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.219478    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.219501    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.219533    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 13:14:53.229958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.222354    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:53.310287 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.310178    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:53.311302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.311276    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:53.311418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.311309    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:53.311418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.311320    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:53.311418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.311346    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.319265 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.319246    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.319397 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.319275    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-122.ec2.internal\": node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.319753 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.319737    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"]
Jul 02 13:14:53.319813 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.319804    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:53.321256 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.321242    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:53.321347 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.321267    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:53.321347 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.321277    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:53.323422 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.323406    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:53.323536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.323522    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.323595 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.323566    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:53.324096 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.324083    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:53.324157 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.324094    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:53.324157 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.324113    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:53.324157 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.324116    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:53.324157 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.324124    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:53.324157 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.324127    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:53.326243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.326229    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.326302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.326263    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 13:14:53.326919 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.326906    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 13:14:53.326983 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.326925    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 13:14:53.326983 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.326935    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeHasSufficientPID"
Jul 02 13:14:53.335801 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.335784    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.339937 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.339919    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b644c5025fc2466a72edbfa99d46c484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b644c5025fc2466a72edbfa99d46c484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.340000 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.339945    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b644c5025fc2466a72edbfa99d46c484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b644c5025fc2466a72edbfa99d46c484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.340000 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.339965    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6c904bbcfca3cc936c3769d2fdfa74-config\") pod \"kube-apiserver-proxy-ip-10-0-133-122.ec2.internal\" (UID: \"cf6c904bbcfca3cc936c3769d2fdfa74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.354567 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.354540    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-122.ec2.internal\" not found" node="ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.359262 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.359244    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-122.ec2.internal\" not found" node="ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.436571 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.436532    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.440865 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.440846    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b644c5025fc2466a72edbfa99d46c484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b644c5025fc2466a72edbfa99d46c484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.440960 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.440873    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b644c5025fc2466a72edbfa99d46c484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b644c5025fc2466a72edbfa99d46c484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.440960 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.440890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6c904bbcfca3cc936c3769d2fdfa74-config\") pod \"kube-apiserver-proxy-ip-10-0-133-122.ec2.internal\" (UID: \"cf6c904bbcfca3cc936c3769d2fdfa74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.440960 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.440933    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf6c904bbcfca3cc936c3769d2fdfa74-config\") pod \"kube-apiserver-proxy-ip-10-0-133-122.ec2.internal\" (UID: \"cf6c904bbcfca3cc936c3769d2fdfa74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.441074 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.440965    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b644c5025fc2466a72edbfa99d46c484-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b644c5025fc2466a72edbfa99d46c484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.441074 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.440978    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b644c5025fc2466a72edbfa99d46c484-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal\" (UID: \"b644c5025fc2466a72edbfa99d46c484\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.537056 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.537027    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.637885 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.637819    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.658234 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.658201    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.663248 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:53.663229    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:53.738908 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.738862    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.839319 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.839289    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:53.939832 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:53.939738    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:54.040211 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:54.040174    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:54.078516 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.078479    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 13:14:54.079007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.078654    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 13:14:54.140519 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:54.140491    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-122.ec2.internal\" not found"
Jul 02 13:14:54.146528 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.146498    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 13:09:53 +0000 UTC" deadline="2027-12-10 23:07:20.327797269 +0000 UTC"
Jul 02 13:14:54.146528 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.146526    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12633h52m26.181274054s"
Jul 02 13:14:54.159050 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.159023    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:54.238328 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.238232    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:54.250386 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.250356    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:54.252007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.251991    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal"
Jul 02 13:14:54.260345 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.260178    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 13:14:54.430208 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:54.430171    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6c904bbcfca3cc936c3769d2fdfa74.slice/crio-60dd2a1bd4c50decea4c9e1014a0a9b9867a0eea0e76370c9b21bf8aa8b68407 WatchSource:0}: Error finding container 60dd2a1bd4c50decea4c9e1014a0a9b9867a0eea0e76370c9b21bf8aa8b68407: Status 404 returned error can't find the container with id 60dd2a1bd4c50decea4c9e1014a0a9b9867a0eea0e76370c9b21bf8aa8b68407
Jul 02 13:14:54.431350 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:54.431315    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb644c5025fc2466a72edbfa99d46c484.slice/crio-f734f140c782df337ddd22948007056a2099d6a978c3ea21a83afa5111b05800 WatchSource:0}: Error finding container f734f140c782df337ddd22948007056a2099d6a978c3ea21a83afa5111b05800: Status 404 returned error can't find the container with id f734f140c782df337ddd22948007056a2099d6a978c3ea21a83afa5111b05800
Jul 02 13:14:54.437355 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.437338    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:14:54.721899 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.721873    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:54.734200 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:54.734162    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:55.056546 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.056451    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 13:14:55.119201 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.119170    2555 apiserver.go:52] "Watching apiserver"
Jul 02 13:14:55.126870 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.126839    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 13:14:55.127324 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.127301    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-c75cn","openshift-multus/network-metrics-daemon-p2b6h","openshift-network-operator/iptables-alerter-fzx5l","openshift-ovn-kubernetes/ovnkube-node-kg5l8","kube-system/global-pull-secret-syncer-5b2xr","kube-system/konnectivity-agent-gmn8c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m","openshift-dns/node-resolver-m5qlt","openshift-image-registry/node-ca-zvw8r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal","openshift-multus/multus-ngm5d","openshift-network-diagnostics/network-check-target-f9hzk","kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal","openshift-cluster-node-tuning-operator/tuned-w2h44"]
Jul 02 13:14:55.130055 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.130030    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.132506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.132478    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:55.132681 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.132649    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:14:55.132918 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.132895    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 13:14:55.133048 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.133017    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n4nnt\""
Jul 02 13:14:55.133154 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.133135    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 13:14:55.133211 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.133135    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 13:14:55.133252 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.133137    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 13:14:55.135101 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.135082    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:55.135202 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.135154    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:14:55.137467 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.137440    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.138649 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.138628    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 13:14:55.140107 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.140085    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 13:14:55.140370 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.140350    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.140473 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.140380    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 13:14:55.141223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.141204    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 13:14:55.141461 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.141442    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 13:14:55.141796 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.141757    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 13:14:55.141878 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.141868    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rdp69\""
Jul 02 13:14:55.141966 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.141948    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 13:14:55.142909 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.142881    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 13:14:55.143571 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.143551    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 13:14:55.143657 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.143639    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.144682 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.144657    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j4x7s\""
Jul 02 13:14:55.146627 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.145028    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 13:14:55.146627 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.146562    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 13:14:55.147339 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.147321    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 13:14:55.147428 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.147361    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qq67g\""
Jul 02 13:14:55.148066 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.148043    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.150595 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150573    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 13:14:55.150909 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150790    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlfl5\" (UniqueName: \"kubernetes.io/projected/7c51b08e-60d3-4455-b38a-74bee9c35ebf-kube-api-access-rlfl5\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:55.150909 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:55.150909 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150858    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-node-log\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.150909 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150885    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-registration-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150909    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/484a8f20-725a-4f94-a670-a40f0db84f7c-cni-binary-copy\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150949    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-kubelet\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150976    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh4xj\" (UniqueName: \"kubernetes.io/projected/484a8f20-725a-4f94-a670-a40f0db84f7c-kube-api-access-bh4xj\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.150989    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26d5q\""
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-etc-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151011    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151035    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-ovn\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151082    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-log-socket\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151107    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-systemd-units\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-cni-bin\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-cni-bin\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151186    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-multus-certs\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151214    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151252    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-run-netns\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-systemd\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151323    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbpbd\" (UniqueName: \"kubernetes.io/projected/1ae190e5-e3a8-4827-8cf2-d01de9ced515-kube-api-access-zbpbd\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151341    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-cni-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151361    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-cnibin\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151386    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-slash\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-var-lib-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151498    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151527    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151537    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-ovnkube-script-lib\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151590    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njthw\" (UniqueName: \"kubernetes.io/projected/16bba2ca-171b-41b4-91c4-d120744ecce2-kube-api-access-njthw\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151615    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-os-release\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151651    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-k8s-cni-cncf-io\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.151730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151682    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16bba2ca-171b-41b4-91c4-d120744ecce2-ovn-node-metrics-cert\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151707    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151759    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-socket-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151810    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-device-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-cni-multus\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151875    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-hostroot\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151911    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-daemon-config\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151938    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-kubelet\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151968    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-cni-netd\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.151994    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-env-overrides\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152026    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-etc-selinux\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-system-cni-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152086    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-netns\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-conf-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-etc-kubernetes\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152173    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-ovnkube-config\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.152522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-sys-fs\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.153132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.152210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-socket-dir-parent\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.154334 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.154066    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.154334 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.154114    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 13:14:55.154334 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.154165    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5nm4r\""
Jul 02 13:14:55.154334 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.154113    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 13:14:55.156694 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.156671    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 13:14:55.156798 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.156701    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 13:14:55.156798 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.156775    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.156921 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.156790    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cbcrc\""
Jul 02 13:14:55.156921 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.156845    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:14:55.157343 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.157200    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 13:14:55.159380 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.159363    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.162047 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.161990    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.162047 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.162019    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzvm8\""
Jul 02 13:14:55.162047 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.162026    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:55.162277 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.162058    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 13:14:55.162277 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.161991    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:55.165024 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.164874    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9pjlx\""
Jul 02 13:14:55.165024 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.164933    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 13:14:55.165148 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.164878    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:14:55.175376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.175356    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8nmgw"
Jul 02 13:14:55.183548 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.183526    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8nmgw"
Jul 02 13:14:55.223942 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.223879    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" event={"ID":"cf6c904bbcfca3cc936c3769d2fdfa74","Type":"ContainerStarted","Data":"60dd2a1bd4c50decea4c9e1014a0a9b9867a0eea0e76370c9b21bf8aa8b68407"}
Jul 02 13:14:55.225073 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.225042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" event={"ID":"b644c5025fc2466a72edbfa99d46c484","Type":"ContainerStarted","Data":"f734f140c782df337ddd22948007056a2099d6a978c3ea21a83afa5111b05800"}
Jul 02 13:14:55.240154 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.240123    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 13:14:55.252993 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.252954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-systemd-units\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-cni-bin\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-cni-bin\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253049    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-multus-certs\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k729c\" (UniqueName: \"kubernetes.io/projected/26b8bcfb-8adf-4773-ae67-670e9899f5bb-kube-api-access-k729c\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253094    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-systemd\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253119    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253120    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-cni-bin\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-cni-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.253186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253212    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysctl-d\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253237    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-multus-certs\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253241    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-lib-modules\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253275    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-systemd\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253307    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736527e9-081f-423d-9b9e-2a38ab79f52a-tmp\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-slash\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253364    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-var-lib-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253392    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-cni-bin\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253437    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-cni-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253491    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-slash\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253506    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-ovnkube-script-lib\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253543    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253558    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-var-lib-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.253663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253620    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-systemd-units\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253665    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njthw\" (UniqueName: \"kubernetes.io/projected/16bba2ca-171b-41b4-91c4-d120744ecce2-kube-api-access-njthw\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-k8s-cni-cncf-io\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253822    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-hostroot\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253834    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-k8s-cni-cncf-io\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253883    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-device-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253913    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-cni-multus\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253974    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-device-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.253976    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-hostroot\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254008    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-daemon-config\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254038    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-cni-multus\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d64dd38-cdb4-4b8c-a31e-39ff013ff70f-agent-certs\") pod \"konnectivity-agent-gmn8c\" (UID: \"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f\") " pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqshn\" (UniqueName: \"kubernetes.io/projected/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-kube-api-access-gqshn\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b39b0b0-9e86-4268-8309-fca12d165a04-kubelet-config\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254312    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-ovnkube-script-lib\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254345    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-kubelet\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.254677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-cni-netd\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254416    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-kubelet\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254459    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-env-overrides\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254495    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-cni-netd\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-system-cni-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254612    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-conf-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-etc-kubernetes\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254652    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-daemon-config\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d64dd38-cdb4-4b8c-a31e-39ff013ff70f-konnectivity-ca\") pod \"konnectivity-agent-gmn8c\" (UID: \"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f\") " pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254715    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-etc-kubernetes\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254722    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-conf-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254747    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-system-cni-dir\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254752    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-serviceca\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254810    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-ovnkube-config\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-kubernetes\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254862    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlfl5\" (UniqueName: \"kubernetes.io/projected/7c51b08e-60d3-4455-b38a-74bee9c35ebf-kube-api-access-rlfl5\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254880    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254904    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-node-log\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.255449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254930    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-kubelet\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254954    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-system-cni-dir\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.254980    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255006    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-tmp-dir\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255029    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-host\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-ovn\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255089    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-node-log\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255106    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-run-ovn\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255094    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-var-lib-kubelet\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255027    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-env-overrides\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255178    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysctl-conf\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-var-lib-kubelet\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-run-netns\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255320    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16bba2ca-171b-41b4-91c4-d120744ecce2-ovnkube-config\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbpbd\" (UniqueName: \"kubernetes.io/projected/1ae190e5-e3a8-4827-8cf2-d01de9ced515-kube-api-access-zbpbd\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.256368 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.255380    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255403    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-cnibin\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255438    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-netns\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.255476    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:14:55.755449293 +0000 UTC m=+2.945778953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255382    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-host-run-netns\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255492    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-host-run-netns\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255481    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-cnibin\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255507    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-os-release\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255565    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-hosts-file\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255569    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-os-release\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255592    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-host\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255625    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-sys\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255669    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16bba2ca-171b-41b4-91c4-d120744ecce2-ovn-node-metrics-cert\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255737    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-socket-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh4xj\" (UniqueName: \"kubernetes.io/projected/484a8f20-725a-4f94-a670-a40f0db84f7c-kube-api-access-bh4xj\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.257222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255788    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89qx\" (UniqueName: \"kubernetes.io/projected/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-kube-api-access-r89qx\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255799    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255811    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-run\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-etc-selinux\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255849    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-socket-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255860    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-tuned\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-sys-fs\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255911    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-socket-dir-parent\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255937    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbb5v\" (UniqueName: \"kubernetes.io/projected/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-kube-api-access-sbb5v\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255954    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-etc-selinux\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255963    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.255994    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-sys-fs\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-registration-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256049    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/484a8f20-725a-4f94-a670-a40f0db84f7c-cni-binary-copy\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256061    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/484a8f20-725a-4f94-a670-a40f0db84f7c-multus-socket-dir-parent\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cnibin\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256076    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1ae190e5-e3a8-4827-8cf2-d01de9ced515-registration-dir\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.258132 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256060    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b39b0b0-9e86-4268-8309-fca12d165a04-dbus\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-iptables-alerter-script\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256169    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-modprobe-d\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysconfig\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256220    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6svk\" (UniqueName: \"kubernetes.io/projected/736527e9-081f-423d-9b9e-2a38ab79f52a-kube-api-access-w6svk\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-etc-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-log-socket\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-os-release\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256315    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-etc-openvswitch\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256327    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16bba2ca-171b-41b4-91c4-d120744ecce2-log-socket\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-host-slash\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256378    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-systemd\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.259062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.256553    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/484a8f20-725a-4f94-a670-a40f0db84f7c-cni-binary-copy\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.260177 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.260151    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16bba2ca-171b-41b4-91c4-d120744ecce2-ovn-node-metrics-cert\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.262055 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.262032    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:55.262173 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.262063    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:55.262173 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.262079    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tpzx7 for pod openshift-network-diagnostics/network-check-target-f9hzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:55.262173 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.262140    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7 podName:a8a20105-d33b-44c0-a6fa-9407191d1dd4 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:55.762120584 +0000 UTC m=+2.952450235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tpzx7" (UniqueName: "kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7") pod "network-check-target-f9hzk" (UID: "a8a20105-d33b-44c0-a6fa-9407191d1dd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:55.263385 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.263354    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njthw\" (UniqueName: \"kubernetes.io/projected/16bba2ca-171b-41b4-91c4-d120744ecce2-kube-api-access-njthw\") pod \"ovnkube-node-kg5l8\" (UID: \"16bba2ca-171b-41b4-91c4-d120744ecce2\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.263523 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.263405    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlfl5\" (UniqueName: \"kubernetes.io/projected/7c51b08e-60d3-4455-b38a-74bee9c35ebf-kube-api-access-rlfl5\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:55.265343 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.265321    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh4xj\" (UniqueName: \"kubernetes.io/projected/484a8f20-725a-4f94-a670-a40f0db84f7c-kube-api-access-bh4xj\") pod \"multus-ngm5d\" (UID: \"484a8f20-725a-4f94-a670-a40f0db84f7c\") " pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.265494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.265473    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbpbd\" (UniqueName: \"kubernetes.io/projected/1ae190e5-e3a8-4827-8cf2-d01de9ced515-kube-api-access-zbpbd\") pod \"aws-ebs-csi-driver-node-pr68m\" (UID: \"1ae190e5-e3a8-4827-8cf2-d01de9ced515\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.357042 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.356937    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-var-lib-kubelet\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.357042 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-hosts-file\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357077    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-host\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357077    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-var-lib-kubelet\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357100    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-sys\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89qx\" (UniqueName: \"kubernetes.io/projected/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-kube-api-access-r89qx\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357149    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-hosts-file\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357156    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-run\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-tuned\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357198    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-sys\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357210    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbb5v\" (UniqueName: \"kubernetes.io/projected/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-kube-api-access-sbb5v\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357240    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-host\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357239    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.357281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cnibin\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357306    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b39b0b0-9e86-4268-8309-fca12d165a04-dbus\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357330    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-iptables-alerter-script\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.357335    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357355    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-modprobe-d\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357381    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysconfig\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.357406    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret podName:9b39b0b0-9e86-4268-8309-fca12d165a04 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:55.857385042 +0000 UTC m=+3.047714700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret") pod "global-pull-secret-syncer-5b2xr" (UID: "9b39b0b0-9e86-4268-8309-fca12d165a04") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357423    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysconfig\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357434    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6svk\" (UniqueName: \"kubernetes.io/projected/736527e9-081f-423d-9b9e-2a38ab79f52a-kube-api-access-w6svk\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357464    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cnibin\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357464    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-os-release\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-host-slash\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357516    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b39b0b0-9e86-4268-8309-fca12d165a04-dbus\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357526    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-systemd\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k729c\" (UniqueName: \"kubernetes.io/projected/26b8bcfb-8adf-4773-ae67-670e9899f5bb-kube-api-access-k729c\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357588    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysctl-d\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-lib-modules\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357721    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736527e9-081f-423d-9b9e-2a38ab79f52a-tmp\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357809    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d64dd38-cdb4-4b8c-a31e-39ff013ff70f-agent-certs\") pod \"konnectivity-agent-gmn8c\" (UID: \"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f\") " pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357816    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-os-release\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqshn\" (UniqueName: \"kubernetes.io/projected/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-kube-api-access-gqshn\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357874    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-modprobe-d\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357726    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cni-binary-copy\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357952    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-systemd\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.357992    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-host-slash\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358035    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b39b0b0-9e86-4268-8309-fca12d165a04-kubelet-config\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d64dd38-cdb4-4b8c-a31e-39ff013ff70f-konnectivity-ca\") pod \"konnectivity-agent-gmn8c\" (UID: \"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f\") " pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-serviceca\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358102    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-iptables-alerter-script\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358149    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysctl-d\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358187    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-lib-modules\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358173    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-run\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358202    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.358890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358223    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-kubernetes\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-system-cni-dir\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358325    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-tmp-dir\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-host\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358372    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b39b0b0-9e86-4268-8309-fca12d165a04-kubelet-config\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358378    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358437    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26b8bcfb-8adf-4773-ae67-670e9899f5bb-system-cni-dir\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysctl-conf\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358671    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-serviceca\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358677    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-kubernetes\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358750    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-sysctl-conf\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358819    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d64dd38-cdb4-4b8c-a31e-39ff013ff70f-konnectivity-ca\") pod \"konnectivity-agent-gmn8c\" (UID: \"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f\") " pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358835    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/736527e9-081f-423d-9b9e-2a38ab79f52a-host\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.358924    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.359021    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-tmp-dir\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.359724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.359027    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/26b8bcfb-8adf-4773-ae67-670e9899f5bb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.361742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.361670    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/736527e9-081f-423d-9b9e-2a38ab79f52a-tmp\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.361742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.361666    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/736527e9-081f-423d-9b9e-2a38ab79f52a-etc-tuned\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.361910 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.361817    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d64dd38-cdb4-4b8c-a31e-39ff013ff70f-agent-certs\") pod \"konnectivity-agent-gmn8c\" (UID: \"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f\") " pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.372897 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.369826    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqshn\" (UniqueName: \"kubernetes.io/projected/db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1-kube-api-access-gqshn\") pod \"node-ca-zvw8r\" (UID: \"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1\") " pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.372897 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.370221    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89qx\" (UniqueName: \"kubernetes.io/projected/70e3cf95-2e8f-4c7c-9a7f-2ac59b982846-kube-api-access-r89qx\") pod \"iptables-alerter-fzx5l\" (UID: \"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846\") " pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.372897 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.372351    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6svk\" (UniqueName: \"kubernetes.io/projected/736527e9-081f-423d-9b9e-2a38ab79f52a-kube-api-access-w6svk\") pod \"tuned-w2h44\" (UID: \"736527e9-081f-423d-9b9e-2a38ab79f52a\") " pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.373561 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.373488    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k729c\" (UniqueName: \"kubernetes.io/projected/26b8bcfb-8adf-4773-ae67-670e9899f5bb-kube-api-access-k729c\") pod \"multus-additional-cni-plugins-c75cn\" (UID: \"26b8bcfb-8adf-4773-ae67-670e9899f5bb\") " pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.374832 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.374805    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbb5v\" (UniqueName: \"kubernetes.io/projected/68762dfb-70dc-4f7c-a38b-6a501e0c52a6-kube-api-access-sbb5v\") pod \"node-resolver-m5qlt\" (UID: \"68762dfb-70dc-4f7c-a38b-6a501e0c52a6\") " pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.442315 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.442269    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngm5d"
Jul 02 13:14:55.452605 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.452567    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:14:55.463263 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.463231    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m"
Jul 02 13:14:55.468898 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.468874    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:14:55.476906 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.476878    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c75cn"
Jul 02 13:14:55.484977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.484949    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m5qlt"
Jul 02 13:14:55.492001 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.491975    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zvw8r"
Jul 02 13:14:55.501370 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.501338    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fzx5l"
Jul 02 13:14:55.506642 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.506616    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w2h44"
Jul 02 13:14:55.761834 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.761708    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:55.762016 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.761850    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:55.762016 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.761940    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:14:56.761916253 +0000 UTC m=+3.952245920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:55.863116 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.863074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:55.863292 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:55.863150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:55.863343 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.863299    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:55.863343 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.863313    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:55.863427 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.863343    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:55.863427 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.863361    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tpzx7 for pod openshift-network-diagnostics/network-check-target-f9hzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:55.863427 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.863383    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret podName:9b39b0b0-9e86-4268-8309-fca12d165a04 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:56.863364396 +0000 UTC m=+4.053694051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret") pod "global-pull-secret-syncer-5b2xr" (UID: "9b39b0b0-9e86-4268-8309-fca12d165a04") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:55.863427 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:55.863410    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7 podName:a8a20105-d33b-44c0-a6fa-9407191d1dd4 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:56.863396875 +0000 UTC m=+4.053726531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tpzx7" (UniqueName: "kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7") pod "network-check-target-f9hzk" (UID: "a8a20105-d33b-44c0-a6fa-9407191d1dd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:56.187487 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:56.187360    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:55 +0000 UTC" deadline="2027-12-31 09:57:02.149928713 +0000 UTC"
Jul 02 13:14:56.187487 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:56.187427    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13124h42m5.962506002s"
Jul 02 13:14:56.220044 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:56.220012    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:56.220211 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.220132    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:14:56.607414 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.607384    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb12a0b5_31c6_45dc_8dc3_0bf8e847e7f1.slice/crio-bbcb6a224fd632e55d711db8785c4e3af6a74643b5e5eeb2dc8459ee92bd3071 WatchSource:0}: Error finding container bbcb6a224fd632e55d711db8785c4e3af6a74643b5e5eeb2dc8459ee92bd3071: Status 404 returned error can't find the container with id bbcb6a224fd632e55d711db8785c4e3af6a74643b5e5eeb2dc8459ee92bd3071
Jul 02 13:14:56.608653 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.608615    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736527e9_081f_423d_9b9e_2a38ab79f52a.slice/crio-c2fab2a6df3fe504d8081e5e60d3be27b2d645baa117c4ad8a7151421cbf68cb WatchSource:0}: Error finding container c2fab2a6df3fe504d8081e5e60d3be27b2d645baa117c4ad8a7151421cbf68cb: Status 404 returned error can't find the container with id c2fab2a6df3fe504d8081e5e60d3be27b2d645baa117c4ad8a7151421cbf68cb
Jul 02 13:14:56.610357 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.610310    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae190e5_e3a8_4827_8cf2_d01de9ced515.slice/crio-30ed7597efa786f7d5a1ec8658a4d98c626d6cbe8f100c6df051a46126c61cf2 WatchSource:0}: Error finding container 30ed7597efa786f7d5a1ec8658a4d98c626d6cbe8f100c6df051a46126c61cf2: Status 404 returned error can't find the container with id 30ed7597efa786f7d5a1ec8658a4d98c626d6cbe8f100c6df051a46126c61cf2
Jul 02 13:14:56.614665 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.614609    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d64dd38_cdb4_4b8c_a31e_39ff013ff70f.slice/crio-40b88c0485c95ed876160f401de88f2ea5e4d8b76806e4d39d2ae08414e2072a WatchSource:0}: Error finding container 40b88c0485c95ed876160f401de88f2ea5e4d8b76806e4d39d2ae08414e2072a: Status 404 returned error can't find the container with id 40b88c0485c95ed876160f401de88f2ea5e4d8b76806e4d39d2ae08414e2072a
Jul 02 13:14:56.615564 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.615538    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b8bcfb_8adf_4773_ae67_670e9899f5bb.slice/crio-ea9aea9da3960cf68c9b5078dcf3c604511b9512ce5832e9306485e4defd862a WatchSource:0}: Error finding container ea9aea9da3960cf68c9b5078dcf3c604511b9512ce5832e9306485e4defd862a: Status 404 returned error can't find the container with id ea9aea9da3960cf68c9b5078dcf3c604511b9512ce5832e9306485e4defd862a
Jul 02 13:14:56.617183 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.617160    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e3cf95_2e8f_4c7c_9a7f_2ac59b982846.slice/crio-9d15cd44d7d44a1ec08a355bfd3eab084022a622f8be85fd8a78357eebf8b875 WatchSource:0}: Error finding container 9d15cd44d7d44a1ec08a355bfd3eab084022a622f8be85fd8a78357eebf8b875: Status 404 returned error can't find the container with id 9d15cd44d7d44a1ec08a355bfd3eab084022a622f8be85fd8a78357eebf8b875
Jul 02 13:14:56.618302 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.618278    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bba2ca_171b_41b4_91c4_d120744ecce2.slice/crio-68c7ca6f4d29ad6342b479eb357969d46f1afcaa17ba205f60a8bdf5149f14ab WatchSource:0}: Error finding container 68c7ca6f4d29ad6342b479eb357969d46f1afcaa17ba205f60a8bdf5149f14ab: Status 404 returned error can't find the container with id 68c7ca6f4d29ad6342b479eb357969d46f1afcaa17ba205f60a8bdf5149f14ab
Jul 02 13:14:56.641304 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.641277    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484a8f20_725a_4f94_a670_a40f0db84f7c.slice/crio-39508f84aaaa07149a12c56601dfd9dd5c7094fdd7933642411288c01d4cd368 WatchSource:0}: Error finding container 39508f84aaaa07149a12c56601dfd9dd5c7094fdd7933642411288c01d4cd368: Status 404 returned error can't find the container with id 39508f84aaaa07149a12c56601dfd9dd5c7094fdd7933642411288c01d4cd368
Jul 02 13:14:56.641883 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:14:56.641863    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68762dfb_70dc_4f7c_a38b_6a501e0c52a6.slice/crio-f41087bac7bef2efb065ea9a0e9f2926eee9a6d97623ee654a64874523cd9946 WatchSource:0}: Error finding container f41087bac7bef2efb065ea9a0e9f2926eee9a6d97623ee654a64874523cd9946: Status 404 returned error can't find the container with id f41087bac7bef2efb065ea9a0e9f2926eee9a6d97623ee654a64874523cd9946
Jul 02 13:14:56.769177 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:56.769080    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:56.769363 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.769269    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:56.769363 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.769359    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.769339803 +0000 UTC m=+5.959669453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:56.870491 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:56.870238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:56.870658 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:56.870530    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:56.870658 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.870445    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:56.870658 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.870578    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:56.870658 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.870593    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tpzx7 for pod openshift-network-diagnostics/network-check-target-f9hzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:56.870913 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.870662    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:56.870913 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.870667    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7 podName:a8a20105-d33b-44c0-a6fa-9407191d1dd4 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.870643746 +0000 UTC m=+6.060973397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tpzx7" (UniqueName: "kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7") pod "network-check-target-f9hzk" (UID: "a8a20105-d33b-44c0-a6fa-9407191d1dd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:56.870913 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:56.870726    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret podName:9b39b0b0-9e86-4268-8309-fca12d165a04 nodeName:}" failed. No retries permitted until 2026-07-02 13:14:58.870707902 +0000 UTC m=+6.061037569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret") pod "global-pull-secret-syncer-5b2xr" (UID: "9b39b0b0-9e86-4268-8309-fca12d165a04") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:57.188304 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.188126    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 13:09:55 +0000 UTC" deadline="2027-11-28 07:19:28.020355778 +0000 UTC"
Jul 02 13:14:57.188304 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.188205    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12330h4m30.832155261s"
Jul 02 13:14:57.223167 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.223114    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:57.223343 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:57.223261    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:14:57.223734 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.223715    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:57.223949 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:57.223925    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:14:57.234137 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.234063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerStarted","Data":"ea9aea9da3960cf68c9b5078dcf3c604511b9512ce5832e9306485e4defd862a"}
Jul 02 13:14:57.239676 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.239580    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w2h44" event={"ID":"736527e9-081f-423d-9b9e-2a38ab79f52a","Type":"ContainerStarted","Data":"c2fab2a6df3fe504d8081e5e60d3be27b2d645baa117c4ad8a7151421cbf68cb"}
Jul 02 13:14:57.241585 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.241515    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmn8c" event={"ID":"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f","Type":"ContainerStarted","Data":"40b88c0485c95ed876160f401de88f2ea5e4d8b76806e4d39d2ae08414e2072a"}
Jul 02 13:14:57.243186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.243097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m" event={"ID":"1ae190e5-e3a8-4827-8cf2-d01de9ced515","Type":"ContainerStarted","Data":"30ed7597efa786f7d5a1ec8658a4d98c626d6cbe8f100c6df051a46126c61cf2"}
Jul 02 13:14:57.247824 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.247759    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zvw8r" event={"ID":"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1","Type":"ContainerStarted","Data":"bbcb6a224fd632e55d711db8785c4e3af6a74643b5e5eeb2dc8459ee92bd3071"}
Jul 02 13:14:57.254596 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.253836    2555 generic.go:356] "Generic (PLEG): container finished" podID="b644c5025fc2466a72edbfa99d46c484" containerID="0a873cd1c5653b68c2156e0975ed7f81b6edc2cae87afedf0225433860715fc1" exitCode=0
Jul 02 13:14:57.254596 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.254046    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" event={"ID":"b644c5025fc2466a72edbfa99d46c484","Type":"ContainerDied","Data":"0a873cd1c5653b68c2156e0975ed7f81b6edc2cae87afedf0225433860715fc1"}
Jul 02 13:14:57.259721 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.259691    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" event={"ID":"cf6c904bbcfca3cc936c3769d2fdfa74","Type":"ContainerStarted","Data":"fe368a9c93ae796e4958ea20a0be3a41269c37c1faa784f5e8f2fb1c8bf4f148"}
Jul 02 13:14:57.262024 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.261986    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m5qlt" event={"ID":"68762dfb-70dc-4f7c-a38b-6a501e0c52a6","Type":"ContainerStarted","Data":"f41087bac7bef2efb065ea9a0e9f2926eee9a6d97623ee654a64874523cd9946"}
Jul 02 13:14:57.263888 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.263808    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngm5d" event={"ID":"484a8f20-725a-4f94-a670-a40f0db84f7c","Type":"ContainerStarted","Data":"39508f84aaaa07149a12c56601dfd9dd5c7094fdd7933642411288c01d4cd368"}
Jul 02 13:14:57.266235 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.266110    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"68c7ca6f4d29ad6342b479eb357969d46f1afcaa17ba205f60a8bdf5149f14ab"}
Jul 02 13:14:57.269524 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.269426    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fzx5l" event={"ID":"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846","Type":"ContainerStarted","Data":"9d15cd44d7d44a1ec08a355bfd3eab084022a622f8be85fd8a78357eebf8b875"}
Jul 02 13:14:57.290959 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:57.290895    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-122.ec2.internal" podStartSLOduration=3.290872412 podStartE2EDuration="3.290872412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:57.290392589 +0000 UTC m=+4.480722297" watchObservedRunningTime="2026-07-02 13:14:57.290872412 +0000 UTC m=+4.481202088"
Jul 02 13:14:58.220426 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:58.219856    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:58.220426 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.220022    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:14:58.289082 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:58.289044    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" event={"ID":"b644c5025fc2466a72edbfa99d46c484","Type":"ContainerStarted","Data":"5784cef85adbe215018292b35c02818ce7bb6d848997ec5850287db4fa57d96a"}
Jul 02 13:14:58.790520 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:58.790456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:58.790846 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.790702    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.790846 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.790790    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.79076936 +0000 UTC m=+9.981099014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:14:58.892039 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:58.891987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:14:58.892256 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:58.892077    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:58.892326 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.892258    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:14:58.892326 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.892278    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:14:58.892326 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.892318    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tpzx7 for pod openshift-network-diagnostics/network-check-target-f9hzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:58.892489 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.892386    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7 podName:a8a20105-d33b-44c0-a6fa-9407191d1dd4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.892365259 +0000 UTC m=+10.082694914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tpzx7" (UniqueName: "kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7") pod "network-check-target-f9hzk" (UID: "a8a20105-d33b-44c0-a6fa-9407191d1dd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:14:58.892489 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.892465    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:58.892608 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:58.892503    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret podName:9b39b0b0-9e86-4268-8309-fca12d165a04 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:02.892489924 +0000 UTC m=+10.082819578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret") pod "global-pull-secret-syncer-5b2xr" (UID: "9b39b0b0-9e86-4268-8309-fca12d165a04") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:14:59.222624 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:59.222479    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:14:59.224183 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:14:59.223315    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:14:59.224183 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:59.223468    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:14:59.224183 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:14:59.224119    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:00.220100 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:00.220060    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:00.220371 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:00.220222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:01.220097 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:01.220062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:01.220097 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:01.220104    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:01.220645 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:01.220234    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:01.220645 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:01.220306    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:02.219849 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:02.219806    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:02.220054 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.219964    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:02.829336 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:02.829294    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:02.829873 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.829496    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:02.829873 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.829573    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:10.829552674 +0000 UTC m=+18.019882347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:02.930747 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:02.930689    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:02.930944 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:02.930785    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:02.930944 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.930924    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:02.931069 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.930990    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret podName:9b39b0b0-9e86-4268-8309-fca12d165a04 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:10.93097161 +0000 UTC m=+18.121301275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret") pod "global-pull-secret-syncer-5b2xr" (UID: "9b39b0b0-9e86-4268-8309-fca12d165a04") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:02.931135 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.931089    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:02.931135 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.931108    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:02.931135 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.931121    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tpzx7 for pod openshift-network-diagnostics/network-check-target-f9hzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:02.931265 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:02.931159    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7 podName:a8a20105-d33b-44c0-a6fa-9407191d1dd4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:10.931147074 +0000 UTC m=+18.121476724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tpzx7" (UniqueName: "kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7") pod "network-check-target-f9hzk" (UID: "a8a20105-d33b-44c0-a6fa-9407191d1dd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:03.224108 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:03.224031    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:03.224423 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:03.224392    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:03.224578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:03.224036    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:03.224955 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:03.224932    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:04.219838 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:04.219801    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:04.220346 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:04.219943    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:05.225412 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:05.224870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:05.225412 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:05.225060    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:05.225412 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:05.225362    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:05.226000 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:05.225486    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:06.219847 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:06.219813    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:06.220001 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:06.219930    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:07.220885 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:07.220843    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:07.221310 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:07.220843    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:07.221310 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:07.220979    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:07.221310 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:07.221096    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:08.220698 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:08.220666    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:08.220918 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:08.220838    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:09.220817 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:09.220779    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:09.220995 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:09.220781    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:09.220995 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:09.220917    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:09.221397 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:09.221029    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:10.220395 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:10.220358    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:10.220607 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.220487    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:10.894368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:10.894331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:10.894802 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.894471    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:10.894802 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.894539    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:26.894520848 +0000 UTC m=+34.084850497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:10.995031 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:10.994989    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:10.995207 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:10.995072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:10.995207 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.995153    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:10.995207 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.995181    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:10.995207 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.995189    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:10.995207 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.995195    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tpzx7 for pod openshift-network-diagnostics/network-check-target-f9hzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:10.995391 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.995249    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7 podName:a8a20105-d33b-44c0-a6fa-9407191d1dd4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:26.995235989 +0000 UTC m=+34.185565638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tpzx7" (UniqueName: "kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7") pod "network-check-target-f9hzk" (UID: "a8a20105-d33b-44c0-a6fa-9407191d1dd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:10.995391 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:10.995271    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret podName:9b39b0b0-9e86-4268-8309-fca12d165a04 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:26.995261144 +0000 UTC m=+34.185590997 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret") pod "global-pull-secret-syncer-5b2xr" (UID: "9b39b0b0-9e86-4268-8309-fca12d165a04") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:11.220235 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:11.220145    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:11.220235 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:11.220164    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:11.220459 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:11.220295    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:11.220459 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:11.220405    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:12.220453 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:12.220421    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:12.220905 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:12.220543    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:13.221122 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:13.221084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:13.221629 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:13.221202    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:13.221629 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:13.221228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:13.221629 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:13.221341    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:14.219934 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:14.219894    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:14.220134 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:14.220030    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:15.219938 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:15.219898    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:15.220407 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:15.220059    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:15.220407 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:15.220117    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:15.220407 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:15.220237    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:16.220808 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.220602    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:16.221275 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:16.220887    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:16.327133 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.327086    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerStarted","Data":"acb9a2bdbaf4e4042d67d95966feba583c23429ca0cd0ec94296baf95f1633e8"}
Jul 02 13:15:16.328894 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.328751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w2h44" event={"ID":"736527e9-081f-423d-9b9e-2a38ab79f52a","Type":"ContainerStarted","Data":"e1f6ec3dd3f656968194fbefaeea0481d85991794d6e88581acce28fcf1baea3"}
Jul 02 13:15:16.330628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.330590    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmn8c" event={"ID":"6d64dd38-cdb4-4b8c-a31e-39ff013ff70f","Type":"ContainerStarted","Data":"ee7cb309edf488a181cf8858ac2b649af5a70a47907d0957e5a5e10d2fd182d3"}
Jul 02 13:15:16.332441 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.332417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m" event={"ID":"1ae190e5-e3a8-4827-8cf2-d01de9ced515","Type":"ContainerStarted","Data":"dd54a14d1014c78e725742cb8447c22c1c440e8110b5bd4657b1e3e4ad47829f"}
Jul 02 13:15:16.334052 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.334016    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zvw8r" event={"ID":"db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1","Type":"ContainerStarted","Data":"630c229923edadcb185bb4864e78d403dac613b18834d7f348349bc5aa92e5d5"}
Jul 02 13:15:16.335614 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.335582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m5qlt" event={"ID":"68762dfb-70dc-4f7c-a38b-6a501e0c52a6","Type":"ContainerStarted","Data":"b40ae465f8df3aa86744bf6b2659acc5b6bfd604fcbc952ce84134f9e1b7ed98"}
Jul 02 13:15:16.337201 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.337179    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngm5d" event={"ID":"484a8f20-725a-4f94-a670-a40f0db84f7c","Type":"ContainerStarted","Data":"206f8c4fd308a8a834f9ac421a72da1857139fe989d473876d5ed19abdf93442"}
Jul 02 13:15:16.339313 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.339292    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:15:16.339717 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.339678    2555 generic.go:356] "Generic (PLEG): container finished" podID="16bba2ca-171b-41b4-91c4-d120744ecce2" containerID="67f9eb2448f78ca2ec297617a431057d52411ba86fad54e73bd18870eadf6e40" exitCode=1
Jul 02 13:15:16.339814 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.339727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerDied","Data":"67f9eb2448f78ca2ec297617a431057d52411ba86fad54e73bd18870eadf6e40"}
Jul 02 13:15:16.339814 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.339790    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"e71b429e10805eca1958266288bee4be66de0eed334eacb3257b3ab46e89ca97"}
Jul 02 13:15:16.349243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.349210    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-122.ec2.internal" podStartSLOduration=22.349199497 podStartE2EDuration="22.349199497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:14:58.305606747 +0000 UTC m=+5.495936425" watchObservedRunningTime="2026-07-02 13:15:16.349199497 +0000 UTC m=+23.539529168"
Jul 02 13:15:16.362357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.362308    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gmn8c" podStartSLOduration=3.05522841 podStartE2EDuration="22.362290688s" totalImagesPullingTime="19.307062278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.36162974 +0000 UTC m=+23.551959411" watchObservedRunningTime="2026-07-02 13:15:16.362290688 +0000 UTC m=+23.552620360"
Jul 02 13:15:16.374792 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.374750    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m5qlt" podStartSLOduration=3.076570626 podStartE2EDuration="22.374731522s" totalImagesPullingTime="19.298160896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.374221314 +0000 UTC m=+23.564550986" watchObservedRunningTime="2026-07-02 13:15:16.374731522 +0000 UTC m=+23.565061198"
Jul 02 13:15:16.387466 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.387425    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zvw8r" podStartSLOduration=3.066280943 podStartE2EDuration="22.387410433s" totalImagesPullingTime="19.32112949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.387166867 +0000 UTC m=+23.577496539" watchObservedRunningTime="2026-07-02 13:15:16.387410433 +0000 UTC m=+23.577740103"
Jul 02 13:15:16.403224 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:16.403183    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ngm5d" podStartSLOduration=4.071857745 podStartE2EDuration="23.40316635s" totalImagesPullingTime="19.331308605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.402733102 +0000 UTC m=+23.593062764" watchObservedRunningTime="2026-07-02 13:15:16.40316635 +0000 UTC m=+23.593496020"
Jul 02 13:15:17.221066 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.220752    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:17.221780 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.220781    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:17.221780 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:17.221143    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:17.221780 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:17.221193    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:17.344343 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.344312    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:15:17.344673 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.344643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"39f158fe467aeeb1d1b8b72f274d40223ff9edc5eec3b92b879013e00a591c74"}
Jul 02 13:15:17.344818 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.344740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"c443bcf6a8f36db84102cda8d88614325066d76dda6c927d478ad9c291e88b78"}
Jul 02 13:15:17.344818 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.344788    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"592ede54eefe72f1773c4e55b687b1c1c588f6a6980dff2923483e63e2a4b54c"}
Jul 02 13:15:17.344818 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.344818    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"9b754074e4f2ddff4bdaaafd66229ea5d9d0ca353932826bf06549aec2180767"}
Jul 02 13:15:17.346083 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.346058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fzx5l" event={"ID":"70e3cf95-2e8f-4c7c-9a7f-2ac59b982846","Type":"ContainerStarted","Data":"29f62edf76166e7365267e058ce4a5742e4b3a9010fdcb9abadc23cb51dfebef"}
Jul 02 13:15:17.347504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.347481    2555 generic.go:356] "Generic (PLEG): container finished" podID="26b8bcfb-8adf-4773-ae67-670e9899f5bb" containerID="acb9a2bdbaf4e4042d67d95966feba583c23429ca0cd0ec94296baf95f1633e8" exitCode=0
Jul 02 13:15:17.347609 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.347570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerDied","Data":"acb9a2bdbaf4e4042d67d95966feba583c23429ca0cd0ec94296baf95f1633e8"}
Jul 02 13:15:17.362227 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.362179    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w2h44" podStartSLOduration=4.0505364 podStartE2EDuration="23.362160404s" totalImagesPullingTime="19.311624004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:16.418564221 +0000 UTC m=+23.608893892" watchObservedRunningTime="2026-07-02 13:15:17.362160404 +0000 UTC m=+24.552490068"
Jul 02 13:15:17.362462 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.362442    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fzx5l" podStartSLOduration=4.078658845 podStartE2EDuration="23.362434455s" totalImagesPullingTime="19.28377561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:17.362231251 +0000 UTC m=+24.552560921" watchObservedRunningTime="2026-07-02 13:15:17.362434455 +0000 UTC m=+24.552764125"
Jul 02 13:15:17.743451 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:17.743423    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 13:15:18.219923 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:18.219888    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:18.220182 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:18.219999    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:18.224986 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:18.224915    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T13:15:17.743441788Z","UUID":"1a6086b5-7610-416b-9eaf-2bc8207b680f","Handler":null,"Name":"","Endpoint":""}
Jul 02 13:15:18.226517 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:18.226497    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 13:15:18.226727 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:18.226538    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 13:15:18.351938 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:18.351895    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m" event={"ID":"1ae190e5-e3a8-4827-8cf2-d01de9ced515","Type":"ContainerStarted","Data":"453fc674eed8e3ddf465329e3fe7ffbde27f9dae4fb8a0e9be1b8d025f7d0ea0"}
Jul 02 13:15:19.219882 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:19.219795    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:19.219882 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:19.219837    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:19.220126 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:19.219954    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:19.220126 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:19.220070    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:19.361472 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:19.361441    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:15:19.361944 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:19.361885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"35cd332699dd1c775f6b3061b2220ee86868b9a2f65e2211ba0aa0294dca5b34"}
Jul 02 13:15:19.983953 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:19.983761    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:15:19.984593 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:19.984570    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:15:20.220271 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:20.220228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:20.220438 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:20.220340    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:20.366852 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:20.366817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m" event={"ID":"1ae190e5-e3a8-4827-8cf2-d01de9ced515","Type":"ContainerStarted","Data":"b95902b2fe67478d136d81e30eb9a6dea87d40f1fe382178e83c6ca121cbdfb4"}
Jul 02 13:15:20.367347 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:20.367133    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:15:20.367655 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:20.367630    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gmn8c"
Jul 02 13:15:20.383696 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:20.383642    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pr68m" podStartSLOduration=4.0920948169999996 podStartE2EDuration="26.383622282s" totalImagesPullingTime="22.291527465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:20.383055441 +0000 UTC m=+27.573385113" watchObservedRunningTime="2026-07-02 13:15:20.383622282 +0000 UTC m=+27.573951953"
Jul 02 13:15:21.220045 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:21.220007    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:21.220243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:21.220007    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:21.220243 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:21.220146    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:21.220361 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:21.220299    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:22.220401 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.220207    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:22.221020 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:22.220519    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:22.374798 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.374770    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:15:22.375226 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.375189    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"a07a24da0ee3d44135bdf9227d3842e73ef6cff4c2844f3eb1b60f0aae4d13d2"}
Jul 02 13:15:22.375467 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.375443    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:15:22.375797 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.375781    2555 scope.go:117] "RemoveContainer" containerID="67f9eb2448f78ca2ec297617a431057d52411ba86fad54e73bd18870eadf6e40"
Jul 02 13:15:22.377168 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.377140    2555 generic.go:356] "Generic (PLEG): container finished" podID="26b8bcfb-8adf-4773-ae67-670e9899f5bb" containerID="83c87a221b7bdbca9628069950f1fe27ec355ccff5f5cb241413e39928b8c867" exitCode=0
Jul 02 13:15:22.377278 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.377200    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerDied","Data":"83c87a221b7bdbca9628069950f1fe27ec355ccff5f5cb241413e39928b8c867"}
Jul 02 13:15:22.392612 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:22.392563    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:15:23.220806 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.220775    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:23.221354 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:23.220898    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:23.221354 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.220911    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:23.221354 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:23.221025    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:23.383886 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.383855    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:15:23.384221 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.384195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" event={"ID":"16bba2ca-171b-41b4-91c4-d120744ecce2","Type":"ContainerStarted","Data":"96fe3d77d4ed7ec17e7e43530bf2948b83f8e03d96b48f574783cfc5e727ecee"}
Jul 02 13:15:23.384523 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.384484    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:15:23.384619 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.384579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:15:23.399658 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.399629    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:15:23.411506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.411458    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8" podStartSLOduration=11.040397443 podStartE2EDuration="30.411437211s" totalImagesPullingTime="19.371039768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:23.411246779 +0000 UTC m=+30.601576451" watchObservedRunningTime="2026-07-02 13:15:23.411437211 +0000 UTC m=+30.601766885"
Jul 02 13:15:23.916729 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.916625    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5b2xr"]
Jul 02 13:15:23.916917 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.916782    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:23.916917 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:23.916898    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:23.920065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.920003    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f9hzk"]
Jul 02 13:15:23.920223 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.920142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:23.920327 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:23.920235    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:23.920847 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.920818    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2b6h"]
Jul 02 13:15:23.920958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:23.920935    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:23.921084 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:23.921055    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:24.389227 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:24.389188    2555 generic.go:356] "Generic (PLEG): container finished" podID="26b8bcfb-8adf-4773-ae67-670e9899f5bb" containerID="cb293d86ac205160266a616e49710c7fbdea60689029e8e1a70d5292a8a9ba87" exitCode=0
Jul 02 13:15:24.389652 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:24.389272    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerDied","Data":"cb293d86ac205160266a616e49710c7fbdea60689029e8e1a70d5292a8a9ba87"}
Jul 02 13:15:25.220466 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:25.220433    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:25.220614 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:25.220557    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:25.220686 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:25.220607    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:25.220765 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:25.220723    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:26.219856 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:26.219642    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:26.220246 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:26.219944    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:26.395902 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:26.395867    2555 generic.go:356] "Generic (PLEG): container finished" podID="26b8bcfb-8adf-4773-ae67-670e9899f5bb" containerID="151f3fac32e2e2fd2ca4aa8ebb5e147a68ff71d61dee1f4eecbdfc15032e9d33" exitCode=0
Jul 02 13:15:26.396053 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:26.395917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerDied","Data":"151f3fac32e2e2fd2ca4aa8ebb5e147a68ff71d61dee1f4eecbdfc15032e9d33"}
Jul 02 13:15:26.912813 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:26.912769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:26.913027 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:26.912945    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:26.913027 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:26.913019    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:15:58.91300221 +0000 UTC m=+66.103331859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 13:15:27.013609 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:27.013559    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:27.013861 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:27.013627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:27.013861 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.013738    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:27.013861 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.013770    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 13:15:27.013861 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.013796    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret podName:9b39b0b0-9e86-4268-8309-fca12d165a04 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:59.013780621 +0000 UTC m=+66.204110270 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret") pod "global-pull-secret-syncer-5b2xr" (UID: "9b39b0b0-9e86-4268-8309-fca12d165a04") : object "kube-system"/"original-pull-secret" not registered
Jul 02 13:15:27.013861 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.013798    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 13:15:27.013861 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.013814    2555 projected.go:196] Error preparing data for projected volume kube-api-access-tpzx7 for pod openshift-network-diagnostics/network-check-target-f9hzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:27.013861 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.013864    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7 podName:a8a20105-d33b-44c0-a6fa-9407191d1dd4 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:59.013849169 +0000 UTC m=+66.204178833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tpzx7" (UniqueName: "kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7") pod "network-check-target-f9hzk" (UID: "a8a20105-d33b-44c0-a6fa-9407191d1dd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 13:15:27.220042 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:27.219954    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:27.220512 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.220067    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:27.220512 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:27.220098    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:27.220512 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:27.220150    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:28.220228 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:28.220192    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:28.220751 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:28.220370    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:15:29.219954 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:29.219915    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:29.220143 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:29.219963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:29.220143 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:29.220098    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9hzk" podUID="a8a20105-d33b-44c0-a6fa-9407191d1dd4"
Jul 02 13:15:29.220279 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:29.220243    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5b2xr" podUID="9b39b0b0-9e86-4268-8309-fca12d165a04"
Jul 02 13:15:30.097077 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.097038    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-122.ec2.internal" event="NodeReady"
Jul 02 13:15:30.097263 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.097211    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 13:15:30.147729 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.147681    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dl649"]
Jul 02 13:15:30.150843 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.150807    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8q44h"]
Jul 02 13:15:30.151026 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.150958    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.152801 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.152772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.153922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.153899    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 13:15:30.153922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.153917    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d2c87\""
Jul 02 13:15:30.154234 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.154217    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 13:15:30.155229 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.155203    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nvh8q\""
Jul 02 13:15:30.155337 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.155318    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 13:15:30.155411 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.155335    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 13:15:30.155642 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.155610    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 13:15:30.159160 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.159135    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8q44h"]
Jul 02 13:15:30.159730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.159657    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dl649"]
Jul 02 13:15:30.220391 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.220354    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:30.223012 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.222983    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:15:30.223162 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.223035    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9zxt\""
Jul 02 13:15:30.236274 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.236229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-config-volume\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.236445 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.236303    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7119527a-4788-4c12-9ca1-198187511679-cert\") pod \"ingress-canary-8q44h\" (UID: \"7119527a-4788-4c12-9ca1-198187511679\") " pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.236445 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.236332    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjvj\" (UniqueName: \"kubernetes.io/projected/7119527a-4788-4c12-9ca1-198187511679-kube-api-access-lqjvj\") pod \"ingress-canary-8q44h\" (UID: \"7119527a-4788-4c12-9ca1-198187511679\") " pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.236445 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.236371    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qq97\" (UniqueName: \"kubernetes.io/projected/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-kube-api-access-7qq97\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.236570 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.236477    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.236570 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.236511    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-tmp-dir\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.337561 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.337522    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-tmp-dir\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.337787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.337583    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-config-volume\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.337787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.337653    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7119527a-4788-4c12-9ca1-198187511679-cert\") pod \"ingress-canary-8q44h\" (UID: \"7119527a-4788-4c12-9ca1-198187511679\") " pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.337787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.337672    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjvj\" (UniqueName: \"kubernetes.io/projected/7119527a-4788-4c12-9ca1-198187511679-kube-api-access-lqjvj\") pod \"ingress-canary-8q44h\" (UID: \"7119527a-4788-4c12-9ca1-198187511679\") " pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.337787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.337740    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qq97\" (UniqueName: \"kubernetes.io/projected/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-kube-api-access-7qq97\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.337787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.337777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.337978 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:30.337885    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:30.337978 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:30.337950    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:30.837931558 +0000 UTC m=+38.028261220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:30.338044 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.338002    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-tmp-dir\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.338375 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.338344    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-config-volume\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.342686 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.342664    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7119527a-4788-4c12-9ca1-198187511679-cert\") pod \"ingress-canary-8q44h\" (UID: \"7119527a-4788-4c12-9ca1-198187511679\") " pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.347724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.347611    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjvj\" (UniqueName: \"kubernetes.io/projected/7119527a-4788-4c12-9ca1-198187511679-kube-api-access-lqjvj\") pod \"ingress-canary-8q44h\" (UID: \"7119527a-4788-4c12-9ca1-198187511679\") " pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.348762 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.348734    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qq97\" (UniqueName: \"kubernetes.io/projected/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-kube-api-access-7qq97\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.471696 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.471646    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8q44h"
Jul 02 13:15:30.842317 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:30.842277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:30.842512 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:30.842462    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:30.842575 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:30.842552    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:31.842527053 +0000 UTC m=+39.032856716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:31.220671 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.220578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:31.221272 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.220602    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:31.224029 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.224002    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:15:31.225167 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.225139    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t5cpn\""
Jul 02 13:15:31.225305 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.225216    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:15:31.225305 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.225234    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:15:31.588251 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.588016    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8q44h"]
Jul 02 13:15:31.596755 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:15:31.596693    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7119527a_4788_4c12_9ca1_198187511679.slice/crio-846c0e2fbe937013c5d22a748be169fd22a0cc6278411887dbdc686b9ec24628 WatchSource:0}: Error finding container 846c0e2fbe937013c5d22a748be169fd22a0cc6278411887dbdc686b9ec24628: Status 404 returned error can't find the container with id 846c0e2fbe937013c5d22a748be169fd22a0cc6278411887dbdc686b9ec24628
Jul 02 13:15:31.851442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:31.851395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:31.851659 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:31.851582    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:31.851746 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:31.851667    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:33.851647763 +0000 UTC m=+41.041977411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:32.415910 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:32.415868    2555 generic.go:356] "Generic (PLEG): container finished" podID="26b8bcfb-8adf-4773-ae67-670e9899f5bb" containerID="761a5d6a6d79a06919677d060f75bab1d49aa6c2ddfac83acf98ed89b1d53b58" exitCode=0
Jul 02 13:15:32.416437 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:32.415982    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerDied","Data":"761a5d6a6d79a06919677d060f75bab1d49aa6c2ddfac83acf98ed89b1d53b58"}
Jul 02 13:15:32.417479 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:32.417451    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8q44h" event={"ID":"7119527a-4788-4c12-9ca1-198187511679","Type":"ContainerStarted","Data":"846c0e2fbe937013c5d22a748be169fd22a0cc6278411887dbdc686b9ec24628"}
Jul 02 13:15:33.424359 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:33.424324    2555 generic.go:356] "Generic (PLEG): container finished" podID="26b8bcfb-8adf-4773-ae67-670e9899f5bb" containerID="939a5b0faab49e00555652840e84d386230f5bc2f905c0f9935e39ae7fec2e46" exitCode=0
Jul 02 13:15:33.424802 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:33.424386    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerDied","Data":"939a5b0faab49e00555652840e84d386230f5bc2f905c0f9935e39ae7fec2e46"}
Jul 02 13:15:33.868488 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:33.868441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:33.868694 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:33.868626    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:33.868783 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:33.868697    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:37.868676486 +0000 UTC m=+45.059006149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:34.430374 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:34.430118    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c75cn" event={"ID":"26b8bcfb-8adf-4773-ae67-670e9899f5bb","Type":"ContainerStarted","Data":"ab59b514e4fc75b74223ad102d0cff1899002fbbc3f25015e55748f63184b2d6"}
Jul 02 13:15:34.431555 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:34.431532    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8q44h" event={"ID":"7119527a-4788-4c12-9ca1-198187511679","Type":"ContainerStarted","Data":"a66ab7c3013d48a46100b7ba609a3c78a97355f2ccdfc1392955561976d20d19"}
Jul 02 13:15:34.451551 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:34.451495    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c75cn" podStartSLOduration=8.754226382 podStartE2EDuration="40.451476559s" totalImagesPullingTime="31.697250177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:34.449862629 +0000 UTC m=+41.640192300" watchObservedRunningTime="2026-07-02 13:15:34.451476559 +0000 UTC m=+41.641806229"
Jul 02 13:15:35.738908 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.738849    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8q44h" podStartSLOduration=3.185654139 podStartE2EDuration="5.738830135s" totalImagesPullingTime="2.553175996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:34.46602714 +0000 UTC m=+41.656356810" watchObservedRunningTime="2026-07-02 13:15:35.738830135 +0000 UTC m=+42.929159784"
Jul 02 13:15:35.739603 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.739584    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"]
Jul 02 13:15:35.772133 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.772092    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"]
Jul 02 13:15:35.772299 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.772194    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:35.776219 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.776193    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 13:15:35.776379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.776238    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 02 13:15:35.776379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.776259    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 13:15:35.776379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.776299    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-z95gn\""
Jul 02 13:15:35.776379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.776299    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 13:15:35.881376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.881336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6\" (UID: \"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:35.881571 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.881389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6xf\" (UniqueName: \"kubernetes.io/projected/2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b-kube-api-access-xj6xf\") pod \"managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6\" (UID: \"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:35.982517 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.982470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6\" (UID: \"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:35.982663 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.982534    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6xf\" (UniqueName: \"kubernetes.io/projected/2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b-kube-api-access-xj6xf\") pod \"managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6\" (UID: \"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:35.986986 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.986962    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6\" (UID: \"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:35.990589 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:35.990528    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6xf\" (UniqueName: \"kubernetes.io/projected/2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b-kube-api-access-xj6xf\") pod \"managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6\" (UID: \"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:36.089863 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:36.089822    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"
Jul 02 13:15:36.238289 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:36.238255    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6"]
Jul 02 13:15:36.242107 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:15:36.242031    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f7fe6ac_f35f_4fe9_9fc6_e0a5ce44b78b.slice/crio-c69652a21f254fabed879c489aa64e926006d7d524ca8d21d2a3b1320a4cde0a WatchSource:0}: Error finding container c69652a21f254fabed879c489aa64e926006d7d524ca8d21d2a3b1320a4cde0a: Status 404 returned error can't find the container with id c69652a21f254fabed879c489aa64e926006d7d524ca8d21d2a3b1320a4cde0a
Jul 02 13:15:36.438290 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:36.438256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6" event={"ID":"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b","Type":"ContainerStarted","Data":"c69652a21f254fabed879c489aa64e926006d7d524ca8d21d2a3b1320a4cde0a"}
Jul 02 13:15:37.896897 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:37.896857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:37.897308 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:37.897023    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:37.897308 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:37.897099    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:15:45.897080345 +0000 UTC m=+53.087410013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:41.453037 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:41.452940    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6" event={"ID":"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b","Type":"ContainerStarted","Data":"504dc2ed23403ca993dce3f211906d96495d7597a31592178ac25af65675a5bc"}
Jul 02 13:15:41.468298 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:41.468227    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6" podStartSLOduration=1.621430017 podStartE2EDuration="6.468210055s" totalImagesPullingTime="4.846780038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:15:41.467993298 +0000 UTC m=+48.658322970" watchObservedRunningTime="2026-07-02 13:15:41.468210055 +0000 UTC m=+48.658539760"
Jul 02 13:15:45.958785 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:45.958741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:15:45.959279 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:45.958915    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:15:45.959279 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:45.958988    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:01.958970106 +0000 UTC m=+69.149299755 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:15:55.412006 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:55.411974    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg5l8"
Jul 02 13:15:58.955029 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:58.954983    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:15:58.957742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:58.957723    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 13:15:58.965907 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:58.965868    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 13:15:58.965993 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:15:58.965982    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:17:02.965960487 +0000 UTC m=+130.156290139 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : secret "metrics-daemon-secret" not found
Jul 02 13:15:59.056140 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.056105    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:59.056236 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.056170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:59.059214 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.059195    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 13:15:59.059313 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.059235    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 13:15:59.068817 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.068790    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b39b0b0-9e86-4268-8309-fca12d165a04-original-pull-secret\") pod \"global-pull-secret-syncer-5b2xr\" (UID: \"9b39b0b0-9e86-4268-8309-fca12d165a04\") " pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:59.069108 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.069094    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 13:15:59.079290 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.079268    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzx7\" (UniqueName: \"kubernetes.io/projected/a8a20105-d33b-44c0-a6fa-9407191d1dd4-kube-api-access-tpzx7\") pod \"network-check-target-f9hzk\" (UID: \"a8a20105-d33b-44c0-a6fa-9407191d1dd4\") " pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:59.135626 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.135597    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t5cpn\""
Jul 02 13:15:59.137594 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.137578    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5b2xr"
Jul 02 13:15:59.143372 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.143356    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:15:59.293096 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.292964    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f9hzk"]
Jul 02 13:15:59.295692 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:15:59.295665    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a20105_d33b_44c0_a6fa_9407191d1dd4.slice/crio-7ba3035f7cb2afdeb0f49953f62bb33bf6fb1c332bce42f56c30df3975873ad7 WatchSource:0}: Error finding container 7ba3035f7cb2afdeb0f49953f62bb33bf6fb1c332bce42f56c30df3975873ad7: Status 404 returned error can't find the container with id 7ba3035f7cb2afdeb0f49953f62bb33bf6fb1c332bce42f56c30df3975873ad7
Jul 02 13:15:59.303440 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.303416    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5b2xr"]
Jul 02 13:15:59.306686 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:15:59.306656    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b39b0b0_9e86_4268_8309_fca12d165a04.slice/crio-6e29c844c10578e67b57ee9a457841f4ec9465acea00409e3050ffa3405e98b4 WatchSource:0}: Error finding container 6e29c844c10578e67b57ee9a457841f4ec9465acea00409e3050ffa3405e98b4: Status 404 returned error can't find the container with id 6e29c844c10578e67b57ee9a457841f4ec9465acea00409e3050ffa3405e98b4
Jul 02 13:15:59.500555 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.500462    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f9hzk" event={"ID":"a8a20105-d33b-44c0-a6fa-9407191d1dd4","Type":"ContainerStarted","Data":"7ba3035f7cb2afdeb0f49953f62bb33bf6fb1c332bce42f56c30df3975873ad7"}
Jul 02 13:15:59.501450 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:15:59.501427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5b2xr" event={"ID":"9b39b0b0-9e86-4268-8309-fca12d165a04","Type":"ContainerStarted","Data":"6e29c844c10578e67b57ee9a457841f4ec9465acea00409e3050ffa3405e98b4"}
Jul 02 13:16:01.981531 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:01.981481    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:16:01.982013 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:16:01.981668    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:16:01.982013 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:16:01.981765    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:16:33.981745096 +0000 UTC m=+101.172074759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:16:04.516499 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:04.516460    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5b2xr" event={"ID":"9b39b0b0-9e86-4268-8309-fca12d165a04","Type":"ContainerStarted","Data":"8bdc168a0416405ddf66d793fe395804bf543f58ec1659043e0dd1196a62407a"}
Jul 02 13:16:04.518087 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:04.518058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f9hzk" event={"ID":"a8a20105-d33b-44c0-a6fa-9407191d1dd4","Type":"ContainerStarted","Data":"7f10dd941cdff1d525591cf3a3148ed1174a1d4855478f0390fbe376841d5f24"}
Jul 02 13:16:04.518228 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:04.518185    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:16:04.530249 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:04.530204    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5b2xr" podStartSLOduration=66.161297568 podStartE2EDuration="1m10.530190006s" totalImagesPullingTime="4.368892438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:04.529981662 +0000 UTC m=+71.720311334" watchObservedRunningTime="2026-07-02 13:16:04.530190006 +0000 UTC m=+71.720519887"
Jul 02 13:16:04.547078 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:04.544791    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f9hzk" podStartSLOduration=67.169286605 podStartE2EDuration="1m11.544772752s" totalImagesPullingTime="4.375486147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:16:04.543875179 +0000 UTC m=+71.734204851" watchObservedRunningTime="2026-07-02 13:16:04.544772752 +0000 UTC m=+71.735102424"
Jul 02 13:16:34.004445 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:34.004400    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:16:34.004918 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:16:34.004566    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:16:34.004918 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:16:34.004654    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:17:38.004636013 +0000 UTC m=+165.194965662 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:16:35.523726 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:16:35.523674    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f9hzk"
Jul 02 13:17:03.013927 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:03.013878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:17:03.014412 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:17:03.014046    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 13:17:03.014412 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:17:03.014129    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs podName:7c51b08e-60d3-4455-b38a-74bee9c35ebf nodeName:}" failed. No retries permitted until 2026-07-02 13:19:05.014112406 +0000 UTC m=+252.204442055 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs") pod "network-metrics-daemon-p2b6h" (UID: "7c51b08e-60d3-4455-b38a-74bee9c35ebf") : secret "metrics-daemon-secret" not found
Jul 02 13:17:33.164450 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:17:33.164398    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dl649" podUID="fc8cbf88-1d0a-4f50-afb3-2b828a2902f8"
Jul 02 13:17:33.230435 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:17:33.230391    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p2b6h" podUID="7c51b08e-60d3-4455-b38a-74bee9c35ebf"
Jul 02 13:17:33.745236 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:33.745149    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dl649"
Jul 02 13:17:38.053561 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:38.053520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:17:38.053985 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:17:38.053643    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 13:17:38.053985 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:17:38.053723    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls podName:fc8cbf88-1d0a-4f50-afb3-2b828a2902f8 nodeName:}" failed. No retries permitted until 2026-07-02 13:19:40.053690542 +0000 UTC m=+287.244020191 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls") pod "dns-default-dl649" (UID: "fc8cbf88-1d0a-4f50-afb3-2b828a2902f8") : secret "dns-default-metrics-tls" not found
Jul 02 13:17:41.766871 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:41.766836    2555 generic.go:356] "Generic (PLEG): container finished" podID="2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b" containerID="504dc2ed23403ca993dce3f211906d96495d7597a31592178ac25af65675a5bc" exitCode=255
Jul 02 13:17:41.767300 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:41.766903    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6" event={"ID":"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b","Type":"ContainerDied","Data":"504dc2ed23403ca993dce3f211906d96495d7597a31592178ac25af65675a5bc"}
Jul 02 13:17:41.772570 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:41.772554    2555 scope.go:117] "RemoveContainer" containerID="504dc2ed23403ca993dce3f211906d96495d7597a31592178ac25af65675a5bc"
Jul 02 13:17:42.771890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:42.771851    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5dfd8ccf48-6prq6" event={"ID":"2f7fe6ac-f35f-4fe9-9fc6-e0a5ce44b78b","Type":"ContainerStarted","Data":"f142efd0a5b53c77cf0f194a6a91e2683cb377563371ac7667c358e0717e8668"}
Jul 02 13:17:45.219917 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:17:45.219828    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:18:26.466421 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.466380    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64b958756c-vc5td"]
Jul 02 13:18:26.468383 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.468365    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5589b49b66-qfwq7"]
Jul 02 13:18:26.468508 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.468489    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.470647 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.470624    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.472578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.472556    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 02 13:18:26.472947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.472928    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jcjbf\""
Jul 02 13:18:26.473059 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.472928    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 02 13:18:26.473198 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.473182    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 13:18:26.473325 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.473294    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 02 13:18:26.473487 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.473473    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 02 13:18:26.474023 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.474001    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 02 13:18:26.474122 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.474046    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 13:18:26.474349 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.474332    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 02 13:18:26.474397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.474349    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 13:18:26.474433 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.474337    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wn2v6\""
Jul 02 13:18:26.479768 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.479751    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 13:18:26.487567 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.487547    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64b958756c-vc5td"]
Jul 02 13:18:26.489194 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.489173    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5589b49b66-qfwq7"]
Jul 02 13:18:26.562025 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.561990    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"]
Jul 02 13:18:26.564145 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.564116    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"]
Jul 02 13:18:26.564272 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.564225    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.566016 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.565997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.566776 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.566754    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 02 13:18:26.566939 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.566816    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 02 13:18:26.567314 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.567297    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 02 13:18:26.567404 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.567334    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vktfl\""
Jul 02 13:18:26.567547 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.567530    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:18:26.568596 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.568581    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 02 13:18:26.568839 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.568826    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 02 13:18:26.569125 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.569112    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 02 13:18:26.569334 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.569320    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xwl2l\""
Jul 02 13:18:26.569641 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.569629    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 02 13:18:26.576467 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.576445    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 02 13:18:26.577896 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.577864    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"]
Jul 02 13:18:26.579126 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.579100    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"]
Jul 02 13:18:26.601547 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601519    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xhk\" (UniqueName: \"kubernetes.io/projected/27cf9350-a187-4386-bd48-3e92edd6810d-kube-api-access-c5xhk\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.601699 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-image-registry-private-configuration\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.601699 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601587    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-stats-auth\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.601699 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601614    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-trusted-ca\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.601699 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601653    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-default-certificate\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.601699 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601692    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf3da4fd-3de2-424a-8106-c214c6d41f6a-ca-trust-extracted\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.601914 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601737    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-certificates\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.601914 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601760    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-installation-pull-secrets\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.601914 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601776    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-bound-sa-token\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.601914 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601794    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.601914 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601825    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.601914 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601860    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.601914 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.601876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjf24\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-kube-api-access-rjf24\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.702199 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702156    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.702418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58538733-ded0-48b6-b964-00c1fa92b611-serving-cert\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.702418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702246    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjf24\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-kube-api-access-rjf24\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.702418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702292    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17c14389-121d-446e-b6a5-dd612b23c463-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.702418 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:26.702343    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:27.202314484 +0000 UTC m=+214.392644155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : configmap references non-existent config key: service-ca.crt
Jul 02 13:18:26.702418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-stats-auth\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.702418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17c14389-121d-446e-b6a5-dd612b23c463-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.702742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702439    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58538733-ded0-48b6-b964-00c1fa92b611-config\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.702742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702476    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-certificates\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.702742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-installation-pull-secrets\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.702742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702523    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-bound-sa-token\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.702742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcg5\" (UniqueName: \"kubernetes.io/projected/58538733-ded0-48b6-b964-00c1fa92b611-kube-api-access-4lcg5\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.702742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702567    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.702742 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702651    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xhk\" (UniqueName: \"kubernetes.io/projected/27cf9350-a187-4386-bd48-3e92edd6810d-kube-api-access-c5xhk\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.703082 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702777    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gpbz\" (UniqueName: \"kubernetes.io/projected/17c14389-121d-446e-b6a5-dd612b23c463-kube-api-access-9gpbz\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.703082 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:26.702731    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:18:26.703082 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.703082 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:26.702853    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:27.202834303 +0000 UTC m=+214.393163963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : secret "router-metrics-certs-default" not found
Jul 02 13:18:26.703246 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.702887    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58538733-ded0-48b6-b964-00c1fa92b611-trusted-ca\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.703588 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:26.703565    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:18:26.703725 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:26.703617    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5589b49b66-qfwq7: secret "image-registry-tls" not found
Jul 02 13:18:26.703725 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.703680    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-image-registry-private-configuration\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.703843 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.703744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-trusted-ca\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.703843 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.703782    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-default-certificate\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.703843 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.703832    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf3da4fd-3de2-424a-8106-c214c6d41f6a-ca-trust-extracted\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.707539 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.704239    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf3da4fd-3de2-424a-8106-c214c6d41f6a-ca-trust-extracted\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.707539 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.703568    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-certificates\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.707539 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:26.705406    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls podName:cf3da4fd-3de2-424a-8106-c214c6d41f6a nodeName:}" failed. No retries permitted until 2026-07-02 13:18:27.20538458 +0000 UTC m=+214.395714229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls") pod "image-registry-5589b49b66-qfwq7" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a") : secret "image-registry-tls" not found
Jul 02 13:18:26.707539 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.705583    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-stats-auth\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.707539 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.705698    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-installation-pull-secrets\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.708518 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.708497    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-trusted-ca\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.709724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.709682    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-image-registry-private-configuration\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.709844 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.709824    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-default-certificate\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.713736 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.713687    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjf24\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-kube-api-access-rjf24\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.714592 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.714556    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-bound-sa-token\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:26.714924 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.714906    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xhk\" (UniqueName: \"kubernetes.io/projected/27cf9350-a187-4386-bd48-3e92edd6810d-kube-api-access-c5xhk\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:26.804787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.804679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58538733-ded0-48b6-b964-00c1fa92b611-serving-cert\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.804787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.804760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17c14389-121d-446e-b6a5-dd612b23c463-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.804787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.804783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17c14389-121d-446e-b6a5-dd612b23c463-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.805078 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.804801    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58538733-ded0-48b6-b964-00c1fa92b611-config\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.805078 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.804826    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcg5\" (UniqueName: \"kubernetes.io/projected/58538733-ded0-48b6-b964-00c1fa92b611-kube-api-access-4lcg5\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.805078 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.804999    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gpbz\" (UniqueName: \"kubernetes.io/projected/17c14389-121d-446e-b6a5-dd612b23c463-kube-api-access-9gpbz\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.805078 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.805061    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58538733-ded0-48b6-b964-00c1fa92b611-trusted-ca\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.805602 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.805573    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58538733-ded0-48b6-b964-00c1fa92b611-config\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.805747 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.805614    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17c14389-121d-446e-b6a5-dd612b23c463-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.805985 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.805964    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58538733-ded0-48b6-b964-00c1fa92b611-trusted-ca\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.807158 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.807130    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17c14389-121d-446e-b6a5-dd612b23c463-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.807258 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.807241    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58538733-ded0-48b6-b964-00c1fa92b611-serving-cert\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.812634 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.812610    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gpbz\" (UniqueName: \"kubernetes.io/projected/17c14389-121d-446e-b6a5-dd612b23c463-kube-api-access-9gpbz\") pod \"kube-storage-version-migrator-operator-79964ff6f9-vszvw\" (UID: \"17c14389-121d-446e-b6a5-dd612b23c463\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.812770 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.812674    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcg5\" (UniqueName: \"kubernetes.io/projected/58538733-ded0-48b6-b964-00c1fa92b611-kube-api-access-4lcg5\") pod \"console-operator-7fdbdf9c99-mcdfs\" (UID: \"58538733-ded0-48b6-b964-00c1fa92b611\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:26.875319 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.875281    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"
Jul 02 13:18:26.882324 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:26.882300    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:27.006683 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:27.006648    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw"]
Jul 02 13:18:27.009963 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:18:27.009935    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c14389_121d_446e_b6a5_dd612b23c463.slice/crio-90fc73f7ed91cebb3b3d941de0215ce335c497f2abf22d4705711febb401127d WatchSource:0}: Error finding container 90fc73f7ed91cebb3b3d941de0215ce335c497f2abf22d4705711febb401127d: Status 404 returned error can't find the container with id 90fc73f7ed91cebb3b3d941de0215ce335c497f2abf22d4705711febb401127d
Jul 02 13:18:27.026037 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:27.026013    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"]
Jul 02 13:18:27.028994 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:18:27.028969    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58538733_ded0_48b6_b964_00c1fa92b611.slice/crio-cc7b580b0aab9045cd23c7b433ad5e6454acdaae0e475479ee8c0bdc6c25597e WatchSource:0}: Error finding container cc7b580b0aab9045cd23c7b433ad5e6454acdaae0e475479ee8c0bdc6c25597e: Status 404 returned error can't find the container with id cc7b580b0aab9045cd23c7b433ad5e6454acdaae0e475479ee8c0bdc6c25597e
Jul 02 13:18:27.207670 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:27.207623    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:27.207890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:27.207698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:27.207890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:27.207739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:27.207890 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:27.207830    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:18:27.207890 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:27.207844    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5589b49b66-qfwq7: secret "image-registry-tls" not found
Jul 02 13:18:27.207890 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:27.207848    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:28.207829907 +0000 UTC m=+215.398159574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : configmap references non-existent config key: service-ca.crt
Jul 02 13:18:27.207890 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:27.207879    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls podName:cf3da4fd-3de2-424a-8106-c214c6d41f6a nodeName:}" failed. No retries permitted until 2026-07-02 13:18:28.207868533 +0000 UTC m=+215.398198183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls") pod "image-registry-5589b49b66-qfwq7" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a") : secret "image-registry-tls" not found
Jul 02 13:18:27.207890 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:27.207829    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:18:27.208125 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:27.207901    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:28.207896237 +0000 UTC m=+215.398225885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : secret "router-metrics-certs-default" not found
Jul 02 13:18:27.911830 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:27.911751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" event={"ID":"58538733-ded0-48b6-b964-00c1fa92b611","Type":"ContainerStarted","Data":"cc7b580b0aab9045cd23c7b433ad5e6454acdaae0e475479ee8c0bdc6c25597e"}
Jul 02 13:18:27.913384 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:27.913321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw" event={"ID":"17c14389-121d-446e-b6a5-dd612b23c463","Type":"ContainerStarted","Data":"90fc73f7ed91cebb3b3d941de0215ce335c497f2abf22d4705711febb401127d"}
Jul 02 13:18:28.215630 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:28.215503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:28.215630 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:28.215620    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:28.215905 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:28.215655    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:28.215905 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:28.215689    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:30.215668334 +0000 UTC m=+217.405997983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : configmap references non-existent config key: service-ca.crt
Jul 02 13:18:28.215905 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:28.215778    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:18:28.215905 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:28.215832    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:30.215819114 +0000 UTC m=+217.406148763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : secret "router-metrics-certs-default" not found
Jul 02 13:18:28.215905 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:28.215780    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:18:28.215905 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:28.215851    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5589b49b66-qfwq7: secret "image-registry-tls" not found
Jul 02 13:18:28.215905 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:28.215873    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls podName:cf3da4fd-3de2-424a-8106-c214c6d41f6a nodeName:}" failed. No retries permitted until 2026-07-02 13:18:30.215866524 +0000 UTC m=+217.406196172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls") pod "image-registry-5589b49b66-qfwq7" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a") : secret "image-registry-tls" not found
Jul 02 13:18:30.233149 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.233036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:30.233149 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.233091    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:30.233149 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.233142    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:30.233628 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:30.233249    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:18:30.233628 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:30.233277    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5589b49b66-qfwq7: secret "image-registry-tls" not found
Jul 02 13:18:30.233628 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:30.233248    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:18:30.233628 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:30.233337    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls podName:cf3da4fd-3de2-424a-8106-c214c6d41f6a nodeName:}" failed. No retries permitted until 2026-07-02 13:18:34.233319765 +0000 UTC m=+221.423649417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls") pod "image-registry-5589b49b66-qfwq7" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a") : secret "image-registry-tls" not found
Jul 02 13:18:30.233628 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:30.233350    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:34.233343908 +0000 UTC m=+221.423673557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : secret "router-metrics-certs-default" not found
Jul 02 13:18:30.233628 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:30.233360    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:34.233354706 +0000 UTC m=+221.423684355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : configmap references non-existent config key: service-ca.crt
Jul 02 13:18:30.788815 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.788782    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9jmz2"]
Jul 02 13:18:30.791258 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.791241    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:30.798775 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.798752    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 13:18:30.799063 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.799044    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rz6tf\""
Jul 02 13:18:30.799179 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.799126    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 13:18:30.800031 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.800018    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 13:18:30.800089 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.800053    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 13:18:30.810869 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.810847    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9jmz2"]
Jul 02 13:18:30.924508 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.924481    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/0.log"
Jul 02 13:18:30.924731 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.924519    2555 generic.go:356] "Generic (PLEG): container finished" podID="58538733-ded0-48b6-b964-00c1fa92b611" containerID="7bbeab2356eb049be4141b866aaeb19a4fe8e106bd1054d9332ebcec863d1ceb" exitCode=255
Jul 02 13:18:30.924731 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.924557    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" event={"ID":"58538733-ded0-48b6-b964-00c1fa92b611","Type":"ContainerDied","Data":"7bbeab2356eb049be4141b866aaeb19a4fe8e106bd1054d9332ebcec863d1ceb"}
Jul 02 13:18:30.924890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.924873    2555 scope.go:117] "RemoveContainer" containerID="7bbeab2356eb049be4141b866aaeb19a4fe8e106bd1054d9332ebcec863d1ceb"
Jul 02 13:18:30.926175 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.926153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw" event={"ID":"17c14389-121d-446e-b6a5-dd612b23c463","Type":"ContainerStarted","Data":"297421307bf61c40781660392c628a8508db4bff8d2d88caf45f50ba6c41271e"}
Jul 02 13:18:30.939344 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.939298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:30.939474 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.939449    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0558b8-7007-427c-b340-f220628ec8d9-data-volume\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:30.940941 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.939491    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:30.940941 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.940009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b0558b8-7007-427c-b340-f220628ec8d9-crio-socket\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:30.940941 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.940055    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8ng\" (UniqueName: \"kubernetes.io/projected/7b0558b8-7007-427c-b340-f220628ec8d9-kube-api-access-2s8ng\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:30.958176 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:30.958124    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw" podStartSLOduration=2.026884939 podStartE2EDuration="4.958105378s" totalImagesPullingTime="2.931220439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:18:30.95786813 +0000 UTC m=+218.148197802" watchObservedRunningTime="2026-07-02 13:18:30.958105378 +0000 UTC m=+218.148435050"
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.040789    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.040915    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.041004    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:31.540982408 +0000 UTC m=+218.731312071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.041049    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0558b8-7007-427c-b340-f220628ec8d9-data-volume\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.041074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.041113    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b0558b8-7007-427c-b340-f220628ec8d9-crio-socket\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.041137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8ng\" (UniqueName: \"kubernetes.io/projected/7b0558b8-7007-427c-b340-f220628ec8d9-kube-api-access-2s8ng\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.041386    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7b0558b8-7007-427c-b340-f220628ec8d9-crio-socket\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.041443    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:18:31.041536 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.041500    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:31.541480823 +0000 UTC m=+218.731810496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : configmap "kube-rbac-proxy" not found
Jul 02 13:18:31.042014 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.041590    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0558b8-7007-427c-b340-f220628ec8d9-data-volume\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.053729 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.053682    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8ng\" (UniqueName: \"kubernetes.io/projected/7b0558b8-7007-427c-b340-f220628ec8d9-kube-api-access-2s8ng\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.545398 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.545350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.545956 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.545415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:31.545956 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.545537    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:31.545956 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.545612    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:32.545595896 +0000 UTC m=+219.735925545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:31.545956 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.545547    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:18:31.545956 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.545693    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:32.545681097 +0000 UTC m=+219.736010756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : configmap "kube-rbac-proxy" not found
Jul 02 13:18:31.930934 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.930905    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:18:31.931328 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.931307    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/0.log"
Jul 02 13:18:31.931446 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.931350    2555 generic.go:356] "Generic (PLEG): container finished" podID="58538733-ded0-48b6-b964-00c1fa92b611" containerID="90f79de7cb6dd6cbe2ea440ce6c4dfdefe9663c1211d93daa88753969855da92" exitCode=255
Jul 02 13:18:31.931494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.931440    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" event={"ID":"58538733-ded0-48b6-b964-00c1fa92b611","Type":"ContainerDied","Data":"90f79de7cb6dd6cbe2ea440ce6c4dfdefe9663c1211d93daa88753969855da92"}
Jul 02 13:18:31.931540 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.931500    2555 scope.go:117] "RemoveContainer" containerID="7bbeab2356eb049be4141b866aaeb19a4fe8e106bd1054d9332ebcec863d1ceb"
Jul 02 13:18:31.931648 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:31.931622    2555 scope.go:117] "RemoveContainer" containerID="90f79de7cb6dd6cbe2ea440ce6c4dfdefe9663c1211d93daa88753969855da92"
Jul 02 13:18:31.931867 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:31.931847    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-mcdfs_openshift-console-operator(58538733-ded0-48b6-b964-00c1fa92b611)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" podUID="58538733-ded0-48b6-b964-00c1fa92b611"
Jul 02 13:18:32.555592 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:32.555551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:32.556027 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:32.555674    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:32.556027 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:32.555685    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:18:32.556027 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:32.555784    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:34.555767336 +0000 UTC m=+221.746096988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : configmap "kube-rbac-proxy" not found
Jul 02 13:18:32.556027 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:32.555795    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:32.556027 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:32.555841    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:34.555826028 +0000 UTC m=+221.746155685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:32.573567 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:32.573543    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m5qlt_68762dfb-70dc-4f7c-a38b-6a501e0c52a6/dns-node-resolver/0.log"
Jul 02 13:18:32.938284 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:32.938257    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:18:32.938605 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:32.938588    2555 scope.go:117] "RemoveContainer" containerID="90f79de7cb6dd6cbe2ea440ce6c4dfdefe9663c1211d93daa88753969855da92"
Jul 02 13:18:32.938783 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:32.938764    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-mcdfs_openshift-console-operator(58538733-ded0-48b6-b964-00c1fa92b611)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" podUID="58538733-ded0-48b6-b964-00c1fa92b611"
Jul 02 13:18:33.974446 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:33.974417    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zvw8r_db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1/node-ca/0.log"
Jul 02 13:18:34.273063 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:34.272955    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:34.273063 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:34.273017    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:34.273306 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:34.273079    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:34.273306 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.273106    2555 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 13:18:34.273306 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.273174    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:42.273157999 +0000 UTC m=+229.463487648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : secret "router-metrics-certs-default" not found
Jul 02 13:18:34.273306 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.273174    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 13:18:34.273306 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.273190    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5589b49b66-qfwq7: secret "image-registry-tls" not found
Jul 02 13:18:34.273306 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.273194    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle podName:27cf9350-a187-4386-bd48-3e92edd6810d nodeName:}" failed. No retries permitted until 2026-07-02 13:18:42.273179625 +0000 UTC m=+229.463509274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle") pod "router-default-64b958756c-vc5td" (UID: "27cf9350-a187-4386-bd48-3e92edd6810d") : configmap references non-existent config key: service-ca.crt
Jul 02 13:18:34.273306 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.273225    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls podName:cf3da4fd-3de2-424a-8106-c214c6d41f6a nodeName:}" failed. No retries permitted until 2026-07-02 13:18:42.273213195 +0000 UTC m=+229.463542849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls") pod "image-registry-5589b49b66-qfwq7" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a") : secret "image-registry-tls" not found
Jul 02 13:18:34.373748 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:34.373701    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8q44h_7119527a-4788-4c12-9ca1-198187511679/serve-healthcheck-canary/0.log"
Jul 02 13:18:34.575285 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:34.575162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:34.575285 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:34.575280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:34.575552 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.575315    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:18:34.575552 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.575365    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:34.575552 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.575398    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:38.575377732 +0000 UTC m=+225.765707383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : configmap "kube-rbac-proxy" not found
Jul 02 13:18:34.575552 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:34.575418    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:38.575409191 +0000 UTC m=+225.765738840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:36.882935 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:36.882898    2555 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:36.883340 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:36.882962    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:36.883340 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:36.883266    2555 scope.go:117] "RemoveContainer" containerID="90f79de7cb6dd6cbe2ea440ce6c4dfdefe9663c1211d93daa88753969855da92"
Jul 02 13:18:36.883451 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:36.883429    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-mcdfs_openshift-console-operator(58538733-ded0-48b6-b964-00c1fa92b611)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" podUID="58538733-ded0-48b6-b964-00c1fa92b611"
Jul 02 13:18:38.609506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:38.609449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:38.609506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:38.609518    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:38.610102 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:38.609600    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 13:18:38.610102 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:38.609671    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:46.609649042 +0000 UTC m=+233.799978696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : configmap "kube-rbac-proxy" not found
Jul 02 13:18:38.610102 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:38.609600    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:38.610102 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:18:38.609762    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls podName:7b0558b8-7007-427c-b340-f220628ec8d9 nodeName:}" failed. No retries permitted until 2026-07-02 13:18:46.60974595 +0000 UTC m=+233.800075599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9jmz2" (UID: "7b0558b8-7007-427c-b340-f220628ec8d9") : secret "insights-runtime-extractor-tls" not found
Jul 02 13:18:42.340935 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.340894    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:42.340935 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.340946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:42.341387 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.340992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:42.341609 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.341590    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27cf9350-a187-4386-bd48-3e92edd6810d-service-ca-bundle\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:42.343344 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.343322    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27cf9350-a187-4386-bd48-3e92edd6810d-metrics-certs\") pod \"router-default-64b958756c-vc5td\" (UID: \"27cf9350-a187-4386-bd48-3e92edd6810d\") " pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:42.343415 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.343379    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"image-registry-5589b49b66-qfwq7\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") " pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:42.378930 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.378903    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:42.384819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.384800    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:42.515493 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.515459    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64b958756c-vc5td"]
Jul 02 13:18:42.519103 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:18:42.519077    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cf9350_a187_4386_bd48_3e92edd6810d.slice/crio-9979e5f1422997983cdf75ab09a82e2a994f9c9f677625e6cef3c0142390a7f2 WatchSource:0}: Error finding container 9979e5f1422997983cdf75ab09a82e2a994f9c9f677625e6cef3c0142390a7f2: Status 404 returned error can't find the container with id 9979e5f1422997983cdf75ab09a82e2a994f9c9f677625e6cef3c0142390a7f2
Jul 02 13:18:42.531805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.531780    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5589b49b66-qfwq7"]
Jul 02 13:18:42.534302 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:18:42.534280    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3da4fd_3de2_424a_8106_c214c6d41f6a.slice/crio-b18c6295719a3d5e2f6ab01d4343b463fa80c1e7aa63484acd7a1034368b9422 WatchSource:0}: Error finding container b18c6295719a3d5e2f6ab01d4343b463fa80c1e7aa63484acd7a1034368b9422: Status 404 returned error can't find the container with id b18c6295719a3d5e2f6ab01d4343b463fa80c1e7aa63484acd7a1034368b9422
Jul 02 13:18:42.975583 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.975547    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64b958756c-vc5td" event={"ID":"27cf9350-a187-4386-bd48-3e92edd6810d","Type":"ContainerStarted","Data":"7a2f5dcde2911cb6da9fdec2dabd52c29f371f12579b372d937d1c32868ad4a3"}
Jul 02 13:18:42.975794 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.975630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64b958756c-vc5td" event={"ID":"27cf9350-a187-4386-bd48-3e92edd6810d","Type":"ContainerStarted","Data":"9979e5f1422997983cdf75ab09a82e2a994f9c9f677625e6cef3c0142390a7f2"}
Jul 02 13:18:42.977116 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.977088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7" event={"ID":"cf3da4fd-3de2-424a-8106-c214c6d41f6a","Type":"ContainerStarted","Data":"08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe"}
Jul 02 13:18:42.977249 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.977131    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7" event={"ID":"cf3da4fd-3de2-424a-8106-c214c6d41f6a","Type":"ContainerStarted","Data":"b18c6295719a3d5e2f6ab01d4343b463fa80c1e7aa63484acd7a1034368b9422"}
Jul 02 13:18:42.977249 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.977151    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:18:42.995257 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:42.995203    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64b958756c-vc5td" podStartSLOduration=16.99518672 podStartE2EDuration="16.99518672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:18:42.994665946 +0000 UTC m=+230.184995617" watchObservedRunningTime="2026-07-02 13:18:42.99518672 +0000 UTC m=+230.185516394"
Jul 02 13:18:43.012406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:43.012359    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7" podStartSLOduration=17.012343802 podStartE2EDuration="17.012343802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:18:43.011760928 +0000 UTC m=+230.202090600" watchObservedRunningTime="2026-07-02 13:18:43.012343802 +0000 UTC m=+230.202673523"
Jul 02 13:18:43.379443 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:43.379402    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:43.382170 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:43.382146    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:43.980781 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:43.980744    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:43.982125 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:43.982104    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64b958756c-vc5td"
Jul 02 13:18:46.675318 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:46.675285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:46.675318 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:46.675322    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:46.675903 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:46.675884    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7b0558b8-7007-427c-b340-f220628ec8d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:46.677583 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:46.677562    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7b0558b8-7007-427c-b340-f220628ec8d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9jmz2\" (UID: \"7b0558b8-7007-427c-b340-f220628ec8d9\") " pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:46.700603 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:46.700570    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9jmz2"
Jul 02 13:18:46.831199 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:46.831164    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9jmz2"]
Jul 02 13:18:46.834223 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:18:46.834193    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0558b8_7007_427c_b340_f220628ec8d9.slice/crio-654ab07b04b390a96b223edfcfc48bc5adce56da7a76915d8d64a7f680454b8f WatchSource:0}: Error finding container 654ab07b04b390a96b223edfcfc48bc5adce56da7a76915d8d64a7f680454b8f: Status 404 returned error can't find the container with id 654ab07b04b390a96b223edfcfc48bc5adce56da7a76915d8d64a7f680454b8f
Jul 02 13:18:46.990520 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:46.990433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jmz2" event={"ID":"7b0558b8-7007-427c-b340-f220628ec8d9","Type":"ContainerStarted","Data":"654ab07b04b390a96b223edfcfc48bc5adce56da7a76915d8d64a7f680454b8f"}
Jul 02 13:18:50.220764 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:50.220724    2555 scope.go:117] "RemoveContainer" containerID="90f79de7cb6dd6cbe2ea440ce6c4dfdefe9663c1211d93daa88753969855da92"
Jul 02 13:18:51.007008 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:51.006901    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jmz2" event={"ID":"7b0558b8-7007-427c-b340-f220628ec8d9","Type":"ContainerStarted","Data":"71a07f81935964a2c06cc2941fee964d2eb1972ac3680a32b38c2dbd87234170"}
Jul 02 13:18:51.008814 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:51.008789    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:18:51.008973 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:51.008873    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" event={"ID":"58538733-ded0-48b6-b964-00c1fa92b611","Type":"ContainerStarted","Data":"b97d76a9e1515f68d023359565ae22a33ce367feff62a2291464511a9a3445fb"}
Jul 02 13:18:51.009218 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:51.009198    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:51.024968 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:51.024919    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs" podStartSLOduration=22.112700435 podStartE2EDuration="25.024907104s" totalImagesPullingTime="2.912206669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:18:51.024341069 +0000 UTC m=+238.214670741" watchObservedRunningTime="2026-07-02 13:18:51.024907104 +0000 UTC m=+238.215236774"
Jul 02 13:18:51.060133 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:51.060106    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-mcdfs"
Jul 02 13:18:52.013490 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:52.013447    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jmz2" event={"ID":"7b0558b8-7007-427c-b340-f220628ec8d9","Type":"ContainerStarted","Data":"de221772e34ce7839477637e341d585270d75748bcc06610d028ab9540d821e9"}
Jul 02 13:18:55.025841 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.025807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9jmz2" event={"ID":"7b0558b8-7007-427c-b340-f220628ec8d9","Type":"ContainerStarted","Data":"26d5bf979c74599621d7dced869088273e03ecf62d3dd0d0b9b7f82e0bf1c050"}
Jul 02 13:18:55.051770 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.051695    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9jmz2" podStartSLOduration=17.828432163 podStartE2EDuration="25.051679211s" totalImagesPullingTime="7.223247048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 13:18:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:18:55.051132309 +0000 UTC m=+242.241461980" watchObservedRunningTime="2026-07-02 13:18:55.051679211 +0000 UTC m=+242.242008937"
Jul 02 13:18:55.128186 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.128155    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-lcvtp"]
Jul 02 13:18:55.131242 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.131221    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-lcvtp"
Jul 02 13:18:55.131395 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.131372    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5589b49b66-qfwq7"]
Jul 02 13:18:55.134843 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.134817    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 02 13:18:55.134976 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.134868    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-x6sld\""
Jul 02 13:18:55.134976 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.134955    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 02 13:18:55.145573 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.145550    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-lcvtp"]
Jul 02 13:18:55.252690 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.252656    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxf78\" (UniqueName: \"kubernetes.io/projected/2d108038-2488-4d4c-917c-3e88960887d3-kube-api-access-zxf78\") pod \"downloads-5499b6fb8f-lcvtp\" (UID: \"2d108038-2488-4d4c-917c-3e88960887d3\") " pod="openshift-console/downloads-5499b6fb8f-lcvtp"
Jul 02 13:18:55.353455 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.353422    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxf78\" (UniqueName: \"kubernetes.io/projected/2d108038-2488-4d4c-917c-3e88960887d3-kube-api-access-zxf78\") pod \"downloads-5499b6fb8f-lcvtp\" (UID: \"2d108038-2488-4d4c-917c-3e88960887d3\") " pod="openshift-console/downloads-5499b6fb8f-lcvtp"
Jul 02 13:18:55.364967 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.364931    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxf78\" (UniqueName: \"kubernetes.io/projected/2d108038-2488-4d4c-917c-3e88960887d3-kube-api-access-zxf78\") pod \"downloads-5499b6fb8f-lcvtp\" (UID: \"2d108038-2488-4d4c-917c-3e88960887d3\") " pod="openshift-console/downloads-5499b6fb8f-lcvtp"
Jul 02 13:18:55.441374 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.441338    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-lcvtp"
Jul 02 13:18:55.570610 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:55.570581    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-lcvtp"]
Jul 02 13:18:55.573104 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:18:55.573073    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d108038_2488_4d4c_917c_3e88960887d3.slice/crio-94d4e4dc0bd4948d6044994332f289e27f43f5619950992b16de03b22fca5b9a WatchSource:0}: Error finding container 94d4e4dc0bd4948d6044994332f289e27f43f5619950992b16de03b22fca5b9a: Status 404 returned error can't find the container with id 94d4e4dc0bd4948d6044994332f289e27f43f5619950992b16de03b22fca5b9a
Jul 02 13:18:56.029984 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:18:56.029949    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-lcvtp" event={"ID":"2d108038-2488-4d4c-917c-3e88960887d3","Type":"ContainerStarted","Data":"94d4e4dc0bd4948d6044994332f289e27f43f5619950992b16de03b22fca5b9a"}
Jul 02 13:19:04.544842 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.544803    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9567bfbb-xt57w"]
Jul 02 13:19:04.547829 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.547797    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.550775 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.550744    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 02 13:19:04.552115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.552025    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dts56\""
Jul 02 13:19:04.552115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.552040    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 02 13:19:04.552115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.552044    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 02 13:19:04.552352 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.552145    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 02 13:19:04.552417 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.552382    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 02 13:19:04.558587 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.558540    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9567bfbb-xt57w"]
Jul 02 13:19:04.625565 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.625525    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-config\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.625769 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.625583    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98cq\" (UniqueName: \"kubernetes.io/projected/fda23ecb-3925-4907-83cf-ffa5350c85b5-kube-api-access-m98cq\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.625769 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.625685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-serving-cert\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.625769 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.625729    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-oauth-config\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.625891 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.625775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-oauth-serving-cert\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.625891 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.625803    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-service-ca\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.726974 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.726931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-config\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.727176 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.726994    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98cq\" (UniqueName: \"kubernetes.io/projected/fda23ecb-3925-4907-83cf-ffa5350c85b5-kube-api-access-m98cq\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.727176 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.727121    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-serving-cert\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.727176 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.727156    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-oauth-config\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.727333 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.727190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-oauth-serving-cert\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.727333 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.727222    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-service-ca\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.727967 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.727938    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-config\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.728108 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.728012    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-oauth-serving-cert\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.728108 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.728071    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-service-ca\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.730190 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.730162    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-oauth-config\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.730325 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.730190    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-serving-cert\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.738553 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.738528    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98cq\" (UniqueName: \"kubernetes.io/projected/fda23ecb-3925-4907-83cf-ffa5350c85b5-kube-api-access-m98cq\") pod \"console-7c9567bfbb-xt57w\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") " pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:04.859755 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:04.859714    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:05.001562 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.001526    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9567bfbb-xt57w"]
Jul 02 13:19:05.004460 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:19:05.004426    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda23ecb_3925_4907_83cf_ffa5350c85b5.slice/crio-60de5d38dbdfa3d94d9dfc701f94d1b22f6856c2273888edb23b21d55c455bbf WatchSource:0}: Error finding container 60de5d38dbdfa3d94d9dfc701f94d1b22f6856c2273888edb23b21d55c455bbf: Status 404 returned error can't find the container with id 60de5d38dbdfa3d94d9dfc701f94d1b22f6856c2273888edb23b21d55c455bbf
Jul 02 13:19:05.030110 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.030072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:19:05.032461 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.032441    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c51b08e-60d3-4455-b38a-74bee9c35ebf-metrics-certs\") pod \"network-metrics-daemon-p2b6h\" (UID: \"7c51b08e-60d3-4455-b38a-74bee9c35ebf\") " pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:19:05.061220 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.061181    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9567bfbb-xt57w" event={"ID":"fda23ecb-3925-4907-83cf-ffa5350c85b5","Type":"ContainerStarted","Data":"60de5d38dbdfa3d94d9dfc701f94d1b22f6856c2273888edb23b21d55c455bbf"}
Jul 02 13:19:05.136947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.136854    2555 patch_prober.go:28] interesting pod/image-registry-5589b49b66-qfwq7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 02 13:19:05.136947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.136914    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7" podUID="cf3da4fd-3de2-424a-8106-c214c6d41f6a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:19:05.323315 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.323282    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k9zxt\""
Jul 02 13:19:05.331591 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.331561    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2b6h"
Jul 02 13:19:05.478235 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:05.478111    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2b6h"]
Jul 02 13:19:05.481245 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:19:05.481212    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c51b08e_60d3_4455_b38a_74bee9c35ebf.slice/crio-5b8260cccc525bd8367f237001428b15495dac7b34c778541e526eeb925a9556 WatchSource:0}: Error finding container 5b8260cccc525bd8367f237001428b15495dac7b34c778541e526eeb925a9556: Status 404 returned error can't find the container with id 5b8260cccc525bd8367f237001428b15495dac7b34c778541e526eeb925a9556
Jul 02 13:19:06.066810 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:06.066762    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2b6h" event={"ID":"7c51b08e-60d3-4455-b38a-74bee9c35ebf","Type":"ContainerStarted","Data":"5b8260cccc525bd8367f237001428b15495dac7b34c778541e526eeb925a9556"}
Jul 02 13:19:07.073673 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:07.073625    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2b6h" event={"ID":"7c51b08e-60d3-4455-b38a-74bee9c35ebf","Type":"ContainerStarted","Data":"bc80629c17bc1d07a5dc953f687f99e759887333ca84d794daa79d478789d73b"}
Jul 02 13:19:07.074192 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:07.073691    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2b6h" event={"ID":"7c51b08e-60d3-4455-b38a-74bee9c35ebf","Type":"ContainerStarted","Data":"a3cfbdf1efdd44d44bd49da59adb9cd53224cd42e9420cc69b454667c59c6222"}
Jul 02 13:19:07.089127 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:07.089061    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p2b6h" podStartSLOduration=253.125230091 podStartE2EDuration="4m14.089040487s" totalImagesPullingTime="963.810396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:07.08824777 +0000 UTC m=+254.278577442" watchObservedRunningTime="2026-07-02 13:19:07.089040487 +0000 UTC m=+254.279370159"
Jul 02 13:19:09.084639 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.084601    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9567bfbb-xt57w" event={"ID":"fda23ecb-3925-4907-83cf-ffa5350c85b5","Type":"ContainerStarted","Data":"157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0"}
Jul 02 13:19:09.102443 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.102394    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9567bfbb-xt57w" podStartSLOduration=1.457611228 podStartE2EDuration="5.102376815s" totalImagesPullingTime="3.644765587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:09.101880532 +0000 UTC m=+256.292210203" watchObservedRunningTime="2026-07-02 13:19:09.102376815 +0000 UTC m=+256.292706486"
Jul 02 13:19:09.363443 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.363352    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"]
Jul 02 13:19:09.365988 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.365964    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.368965 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.368939    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 02 13:19:09.368965 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.368955    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 13:19:09.370362 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.370335    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qpqrx\""
Jul 02 13:19:09.370506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.370374    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 13:19:09.370506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.370447    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 13:19:09.370506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.370463    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:19:09.377884 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.377862    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"]
Jul 02 13:19:09.380801 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.380777    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-j2xzr"]
Jul 02 13:19:09.383219 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.383147    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.385879 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.385829    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 02 13:19:09.385879 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.385858    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b4vnh\""
Jul 02 13:19:09.386053 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.385948    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 02 13:19:09.386322 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.386301    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 02 13:19:09.394895 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.394867    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-96v2m"]
Jul 02 13:19:09.400826 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.400471    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-j2xzr"]
Jul 02 13:19:09.400826 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.400727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.403968 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.403933    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 13:19:09.404138 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.403934    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 13:19:09.405464 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.405444    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 13:19:09.405937 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.405916    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vl9hz\""
Jul 02 13:19:09.468281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468240    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgc2\" (UniqueName: \"kubernetes.io/projected/9c24256d-7842-44b0-9f28-153d5e5c1176-kube-api-access-9tgc2\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.468281 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468285    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.468550 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468368    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.468550 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fkz\" (UniqueName: \"kubernetes.io/projected/4d17e46d-255d-4497-b605-445ed68e4cc2-kube-api-access-p7fkz\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.468550 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-root\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.468550 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468455    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d17e46d-255d-4497-b605-445ed68e4cc2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.468550 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468510    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.468763 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468584    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6fx\" (UniqueName: \"kubernetes.io/projected/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-api-access-kf6fx\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.468763 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468617    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c24256d-7842-44b0-9f28-153d5e5c1176-metrics-client-ca\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.468763 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-metrics-client-ca\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.468763 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-accelerators-collector-config\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.468763 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468747    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d17e46d-255d-4497-b605-445ed68e4cc2-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.468763 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468761    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-sys\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.469004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.469004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-wtmp\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.469004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468863    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-textfile\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.469004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-tls\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.469004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468949    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d17e46d-255d-4497-b605-445ed68e4cc2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.469004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.468992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.570004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.569964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c24256d-7842-44b0-9f28-153d5e5c1176-metrics-client-ca\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570200 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570011    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-metrics-client-ca\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.570200 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-accelerators-collector-config\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570200 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d17e46d-255d-4497-b605-445ed68e4cc2-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.570369 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570269    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-sys\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570369 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.570369 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570367    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-wtmp\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570518 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-textfile\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570518 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570403    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-sys\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570518 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-tls\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570518 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570472    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d17e46d-255d-4497-b605-445ed68e4cc2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.570518 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgc2\" (UniqueName: \"kubernetes.io/projected/9c24256d-7842-44b0-9f28-153d5e5c1176-kube-api-access-9tgc2\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570557    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-wtmp\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570578    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570619    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fkz\" (UniqueName: \"kubernetes.io/projected/4d17e46d-255d-4497-b605-445ed68e4cc2-kube-api-access-p7fkz\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-root\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570738    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d17e46d-255d-4497-b605-445ed68e4cc2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570769    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6fx\" (UniqueName: \"kubernetes.io/projected/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-api-access-kf6fx\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.570819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570817    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-metrics-client-ca\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.571390 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570833    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c24256d-7842-44b0-9f28-153d5e5c1176-metrics-client-ca\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.571390 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570844    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-accelerators-collector-config\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.571390 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:19:09.570944    2555 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 02 13:19:09.571390 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.570986    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d17e46d-255d-4497-b605-445ed68e4cc2-metrics-client-ca\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.571390 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:19:09.571002    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-tls podName:01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f nodeName:}" failed. No retries permitted until 2026-07-02 13:19:10.070984771 +0000 UTC m=+257.261314419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-j2xzr" (UID: "01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f") : secret "kube-state-metrics-tls" not found
Jul 02 13:19:09.571390 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.571089    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-textfile\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.571390 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.571140    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c24256d-7842-44b0-9f28-153d5e5c1176-root\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.571753 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.571489    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.571753 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.571623    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.573895 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.573861    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d17e46d-255d-4497-b605-445ed68e4cc2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.574030 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.573894    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d17e46d-255d-4497-b605-445ed68e4cc2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.574030 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.573946    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.574183 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.574162    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.574994 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.574964    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c24256d-7842-44b0-9f28-153d5e5c1176-node-exporter-tls\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.582719 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.582686    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fkz\" (UniqueName: \"kubernetes.io/projected/4d17e46d-255d-4497-b605-445ed68e4cc2-kube-api-access-p7fkz\") pod \"openshift-state-metrics-57f99c8b49-tvw9b\" (UID: \"4d17e46d-255d-4497-b605-445ed68e4cc2\") " pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.582995 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.582971    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgc2\" (UniqueName: \"kubernetes.io/projected/9c24256d-7842-44b0-9f28-153d5e5c1176-kube-api-access-9tgc2\") pod \"node-exporter-96v2m\" (UID: \"9c24256d-7842-44b0-9f28-153d5e5c1176\") " pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:09.583075 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.583058    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6fx\" (UniqueName: \"kubernetes.io/projected/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-api-access-kf6fx\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:09.677482 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.677387    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"
Jul 02 13:19:09.713667 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:09.713626    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-96v2m"
Jul 02 13:19:10.075795 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:10.075678    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:10.078351 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:10.078324    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-j2xzr\" (UID: \"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f\") " pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:10.295295 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:10.295028    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr"
Jul 02 13:19:13.673656 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.673626    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"]
Jul 02 13:19:13.675976 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.675948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.680609 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.680431    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 02 13:19:13.680609 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.680445    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 02 13:19:13.680856 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.680781    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6tnf4\""
Jul 02 13:19:13.680856 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.680814    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d6d9a1tp4h70j\""
Jul 02 13:19:13.680966 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.680858    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 02 13:19:13.681168 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.681134    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 02 13:19:13.687350 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.687322    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"]
Jul 02 13:19:13.810183 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.810137    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3269a7ce-84b9-4f90-99e4-7c2730e9b336-audit-log\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.810392 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.810198    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-secret-metrics-server-client-certs\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.810392 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.810290    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxm2\" (UniqueName: \"kubernetes.io/projected/3269a7ce-84b9-4f90-99e4-7c2730e9b336-kube-api-access-dsxm2\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.810392 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.810336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3269a7ce-84b9-4f90-99e4-7c2730e9b336-metrics-server-audit-profiles\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.810578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.810416    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3269a7ce-84b9-4f90-99e4-7c2730e9b336-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.810578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.810466    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-secret-metrics-server-tls\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.810578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.810489    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-client-ca-bundle\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.911460 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3269a7ce-84b9-4f90-99e4-7c2730e9b336-audit-log\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.911665 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911473    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-secret-metrics-server-client-certs\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.911665 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911508    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxm2\" (UniqueName: \"kubernetes.io/projected/3269a7ce-84b9-4f90-99e4-7c2730e9b336-kube-api-access-dsxm2\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.911665 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911540    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3269a7ce-84b9-4f90-99e4-7c2730e9b336-metrics-server-audit-profiles\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.911665 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911606    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3269a7ce-84b9-4f90-99e4-7c2730e9b336-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.911903 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-secret-metrics-server-tls\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.911903 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911778    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-client-ca-bundle\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.912012 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.911940    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3269a7ce-84b9-4f90-99e4-7c2730e9b336-audit-log\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.912462 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.912427    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3269a7ce-84b9-4f90-99e4-7c2730e9b336-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.912749 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.912726    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3269a7ce-84b9-4f90-99e4-7c2730e9b336-metrics-server-audit-profiles\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.914610 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.914580    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-client-ca-bundle\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.914719 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.914582    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-secret-metrics-server-client-certs\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.914719 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.914666    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3269a7ce-84b9-4f90-99e4-7c2730e9b336-secret-metrics-server-tls\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.919438 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.919418    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxm2\" (UniqueName: \"kubernetes.io/projected/3269a7ce-84b9-4f90-99e4-7c2730e9b336-kube-api-access-dsxm2\") pod \"metrics-server-d6d8c96cf-zl9xk\" (UID: \"3269a7ce-84b9-4f90-99e4-7c2730e9b336\") " pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:13.992377 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:13.992257    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:14.342992 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.342950    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-744f7db998-t6tr9"]
Jul 02 13:19:14.345601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.345577    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.356672 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.356436    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 02 13:19:14.358618 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.358594    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744f7db998-t6tr9"]
Jul 02 13:19:14.415835 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.415798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-serving-cert\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.415835 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.415841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-oauth-config\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.416093 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.415865    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-oauth-serving-cert\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.416093 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.415886    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfbt\" (UniqueName: \"kubernetes.io/projected/5f0016e2-2afa-4dba-94af-46dcc6aff297-kube-api-access-xnfbt\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.416093 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.415971    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-service-ca\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.416093 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.416060    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-trusted-ca-bundle\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.416255 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.416128    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-config\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.517184 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.517150    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-oauth-config\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.517184 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.517186    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-oauth-serving-cert\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.517454 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.517207    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfbt\" (UniqueName: \"kubernetes.io/projected/5f0016e2-2afa-4dba-94af-46dcc6aff297-kube-api-access-xnfbt\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.517454 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.517267    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-service-ca\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.517454 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.517321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-trusted-ca-bundle\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.517454 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.517401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-config\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.517661 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.517450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-serving-cert\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.518259 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.518200    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-oauth-serving-cert\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.518396 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.518313    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-config\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.518472 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.518442    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-trusted-ca-bundle\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.518987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.518944    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-service-ca\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.520318 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.520294    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-oauth-config\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.520418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.520325    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-serving-cert\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.525145 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.525125    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfbt\" (UniqueName: \"kubernetes.io/projected/5f0016e2-2afa-4dba-94af-46dcc6aff297-kube-api-access-xnfbt\") pod \"console-744f7db998-t6tr9\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") " pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.658442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.658351    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:14.860183 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.860141    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:14.860666 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.860220    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:14.866074 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:14.866049    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:15.112452 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:15.112421    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:15.137302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:15.137274    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:19:16.513192 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:19:16.512962    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c24256d_7842_44b0_9f28_153d5e5c1176.slice/crio-28388870f96f1961915b51d0d613f6852d5343009acb0e64f953a8dacde7a441 WatchSource:0}: Error finding container 28388870f96f1961915b51d0d613f6852d5343009acb0e64f953a8dacde7a441: Status 404 returned error can't find the container with id 28388870f96f1961915b51d0d613f6852d5343009acb0e64f953a8dacde7a441
Jul 02 13:19:16.674985 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:16.674080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"]
Jul 02 13:19:16.693557 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:16.693345    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-j2xzr"]
Jul 02 13:19:16.696513 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:19:16.696472    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fa2353_6942_4c5f_b3d8_7e6e6cbbb64f.slice/crio-ddad1c20395930ad88d9044ec8c2aac1b919ea688629dbd370ff780fbfcc8d12 WatchSource:0}: Error finding container ddad1c20395930ad88d9044ec8c2aac1b919ea688629dbd370ff780fbfcc8d12: Status 404 returned error can't find the container with id ddad1c20395930ad88d9044ec8c2aac1b919ea688629dbd370ff780fbfcc8d12
Jul 02 13:19:16.719746 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:16.719693    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b"]
Jul 02 13:19:16.720938 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:16.720915    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-744f7db998-t6tr9"]
Jul 02 13:19:16.723756 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:19:16.723728    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0016e2_2afa_4dba_94af_46dcc6aff297.slice/crio-6a20f3eae0ce9851ff5e5bcae4d04cac6561d37a4c156f7fd455dc3d8aac4c02 WatchSource:0}: Error finding container 6a20f3eae0ce9851ff5e5bcae4d04cac6561d37a4c156f7fd455dc3d8aac4c02: Status 404 returned error can't find the container with id 6a20f3eae0ce9851ff5e5bcae4d04cac6561d37a4c156f7fd455dc3d8aac4c02
Jul 02 13:19:16.725306 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:19:16.725286    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d17e46d_255d_4497_b605_445ed68e4cc2.slice/crio-01e19794e9f7d4ce001f1b7c478a9c25308912178315de62cd854c6b2e213f36 WatchSource:0}: Error finding container 01e19794e9f7d4ce001f1b7c478a9c25308912178315de62cd854c6b2e213f36: Status 404 returned error can't find the container with id 01e19794e9f7d4ce001f1b7c478a9c25308912178315de62cd854c6b2e213f36
Jul 02 13:19:17.121276 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.121103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b" event={"ID":"4d17e46d-255d-4497-b605-445ed68e4cc2","Type":"ContainerStarted","Data":"76e66dcce1f89b30baa74c86cd99bb80e20d46ce70f5685f740e60df8340be2e"}
Jul 02 13:19:17.121276 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.121185    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b" event={"ID":"4d17e46d-255d-4497-b605-445ed68e4cc2","Type":"ContainerStarted","Data":"e4bf21a5d8588200719eeafbc466c1d9be6b6b008c322fe704c3b9a97daf9852"}
Jul 02 13:19:17.121276 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.121218    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b" event={"ID":"4d17e46d-255d-4497-b605-445ed68e4cc2","Type":"ContainerStarted","Data":"01e19794e9f7d4ce001f1b7c478a9c25308912178315de62cd854c6b2e213f36"}
Jul 02 13:19:17.123579 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.123520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7db998-t6tr9" event={"ID":"5f0016e2-2afa-4dba-94af-46dcc6aff297","Type":"ContainerStarted","Data":"e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0"}
Jul 02 13:19:17.123785 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.123583    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7db998-t6tr9" event={"ID":"5f0016e2-2afa-4dba-94af-46dcc6aff297","Type":"ContainerStarted","Data":"6a20f3eae0ce9851ff5e5bcae4d04cac6561d37a4c156f7fd455dc3d8aac4c02"}
Jul 02 13:19:17.125599 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.125532    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk" event={"ID":"3269a7ce-84b9-4f90-99e4-7c2730e9b336","Type":"ContainerStarted","Data":"a1a9b0ea23e376fd70ef2f2738add555d880f2ac864277627659339cab531bb2"}
Jul 02 13:19:17.128616 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.128580    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr" event={"ID":"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f","Type":"ContainerStarted","Data":"ddad1c20395930ad88d9044ec8c2aac1b919ea688629dbd370ff780fbfcc8d12"}
Jul 02 13:19:17.131341 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.131303    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-lcvtp" event={"ID":"2d108038-2488-4d4c-917c-3e88960887d3","Type":"ContainerStarted","Data":"59359a4cc2c5eef297c16db4f9b9e87b43aace85c256784e0badf7702356c517"}
Jul 02 13:19:17.132612 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.132531    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-lcvtp"
Jul 02 13:19:17.134377 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.134324    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96v2m" event={"ID":"9c24256d-7842-44b0-9f28-153d5e5c1176","Type":"ContainerStarted","Data":"28388870f96f1961915b51d0d613f6852d5343009acb0e64f953a8dacde7a441"}
Jul 02 13:19:17.142537 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.142479    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-744f7db998-t6tr9" podStartSLOduration=3.142459761 podStartE2EDuration="3.142459761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:17.142374871 +0000 UTC m=+264.332704537" watchObservedRunningTime="2026-07-02 13:19:17.142459761 +0000 UTC m=+264.332789434"
Jul 02 13:19:17.142746 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.142724    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-lcvtp"
Jul 02 13:19:17.160128 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:17.158280    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-lcvtp" podStartSLOduration=1.1530088 podStartE2EDuration="22.158243967s" totalImagesPullingTime="21.005235167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:17.157931469 +0000 UTC m=+264.348261141" watchObservedRunningTime="2026-07-02 13:19:17.158243967 +0000 UTC m=+264.348573664"
Jul 02 13:19:18.147294 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:18.145070    2555 generic.go:356] "Generic (PLEG): container finished" podID="9c24256d-7842-44b0-9f28-153d5e5c1176" containerID="e40ac67084680dfcbdb72075d26a81b1e9167e12db89f7eb5a2fce7e90dc8bfb" exitCode=0
Jul 02 13:19:18.147294 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:18.146521    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96v2m" event={"ID":"9c24256d-7842-44b0-9f28-153d5e5c1176","Type":"ContainerDied","Data":"e40ac67084680dfcbdb72075d26a81b1e9167e12db89f7eb5a2fce7e90dc8bfb"}
Jul 02 13:19:19.152201 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:19.152162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96v2m" event={"ID":"9c24256d-7842-44b0-9f28-153d5e5c1176","Type":"ContainerStarted","Data":"9a979d2ff105063b9c0736d8f01c24e4afac2c3ed91cd6c24789f0f0439c4715"}
Jul 02 13:19:20.154097 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.154034    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7" podUID="cf3da4fd-3de2-424a-8106-c214c6d41f6a" containerName="registry" containerID="cri-o://08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe" gracePeriod=30
Jul 02 13:19:20.463288 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.462939    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:19:20.589423 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.589394    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-trusted-ca\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.590021 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.590003    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-bound-sa-token\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.590247 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.590232    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf3da4fd-3de2-424a-8106-c214c6d41f6a-ca-trust-extracted\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.590895 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.590373    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-installation-pull-secrets\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.591040 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.591023    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-certificates\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.591162 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.591148    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjf24\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-kube-api-access-rjf24\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.591288 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.591275    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.591418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.591405    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-image-registry-private-configuration\") pod \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\" (UID: \"cf3da4fd-3de2-424a-8106-c214c6d41f6a\") "
Jul 02 13:19:20.592681 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.589951    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:19:20.592956 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.592922    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:19:20.594392 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.594367    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:19:20.595172 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.595149    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:19:20.595426 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.595395    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:19:20.597730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.597632    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-kube-api-access-rjf24" (OuterVolumeSpecName: "kube-api-access-rjf24") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "kube-api-access-rjf24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:19:20.598825 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.597943    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:19:20.602487 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.602465    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3da4fd-3de2-424a-8106-c214c6d41f6a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cf3da4fd-3de2-424a-8106-c214c6d41f6a" (UID: "cf3da4fd-3de2-424a-8106-c214c6d41f6a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:19:20.692667 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692627    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-image-registry-private-configuration\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:20.692667 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692665    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-trusted-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:20.692922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692680    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-bound-sa-token\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:20.692922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692695    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf3da4fd-3de2-424a-8106-c214c6d41f6a-ca-trust-extracted\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:20.692922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692734    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf3da4fd-3de2-424a-8106-c214c6d41f6a-installation-pull-secrets\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:20.692922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692749    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-certificates\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:20.692922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692759    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjf24\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-kube-api-access-rjf24\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:20.692922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:20.692768    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf3da4fd-3de2-424a-8106-c214c6d41f6a-registry-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:21.163013 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.162974    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk" event={"ID":"3269a7ce-84b9-4f90-99e4-7c2730e9b336","Type":"ContainerStarted","Data":"240e1f5c9e435428f86f034f1faf172faf8df598ca235dd28ada0dd89f644561"}
Jul 02 13:19:21.164622 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.164591    2555 generic.go:356] "Generic (PLEG): container finished" podID="cf3da4fd-3de2-424a-8106-c214c6d41f6a" containerID="08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe" exitCode=0
Jul 02 13:19:21.164785 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.164632    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7"
Jul 02 13:19:21.164785 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.164668    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7" event={"ID":"cf3da4fd-3de2-424a-8106-c214c6d41f6a","Type":"ContainerDied","Data":"08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe"}
Jul 02 13:19:21.164785 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.164739    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5589b49b66-qfwq7" event={"ID":"cf3da4fd-3de2-424a-8106-c214c6d41f6a","Type":"ContainerDied","Data":"b18c6295719a3d5e2f6ab01d4343b463fa80c1e7aa63484acd7a1034368b9422"}
Jul 02 13:19:21.164785 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.164771    2555 scope.go:117] "RemoveContainer" containerID="08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe"
Jul 02 13:19:21.167525 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.167497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr" event={"ID":"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f","Type":"ContainerStarted","Data":"be4d37a6e45995582ca815f6561db95c06beabff02a366f05032375947ac6f5e"}
Jul 02 13:19:21.167640 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.167554    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr" event={"ID":"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f","Type":"ContainerStarted","Data":"19027de06ed6e40c71a453487ddf8116fc9182a417a4b05046c3255a0663ce2f"}
Jul 02 13:19:21.167640 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.167583    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr" event={"ID":"01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f","Type":"ContainerStarted","Data":"5a48d98892237cfe6f92e7a4dac80725d701017b88c9d70473acbafb4f2511a5"}
Jul 02 13:19:21.170911 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.170686    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-96v2m" event={"ID":"9c24256d-7842-44b0-9f28-153d5e5c1176","Type":"ContainerStarted","Data":"39603e43703e93ef59d25b5c4a53ad0d5319f81e4295aedb79884cb8a1b43363"}
Jul 02 13:19:21.176012 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.175907    2555 scope.go:117] "RemoveContainer" containerID="08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe"
Jul 02 13:19:21.176524 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:19:21.176322    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe\": container with ID starting with 08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe not found: ID does not exist" containerID="08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe"
Jul 02 13:19:21.176524 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.176360    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe"} err="failed to get container status \"08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe\": rpc error: code = NotFound desc = could not find container \"08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe\": container with ID starting with 08cdd41712ab995a90def336665bdb523246980f8a6a78f04e104a5a4582b0fe not found: ID does not exist"
Jul 02 13:19:21.176659 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.176579    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b" event={"ID":"4d17e46d-255d-4497-b605-445ed68e4cc2","Type":"ContainerStarted","Data":"31658061a8445c3c460d5688c402756cc9aaf3b0e6672b22efc08d79852b7dbf"}
Jul 02 13:19:21.183675 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.183627    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk" podStartSLOduration=4.749273454 podStartE2EDuration="8.183609722s" totalImagesPullingTime="3.434336268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:21.181194071 +0000 UTC m=+268.371523743" watchObservedRunningTime="2026-07-02 13:19:21.183609722 +0000 UTC m=+268.373939375"
Jul 02 13:19:21.200988 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.200926    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-57f99c8b49-tvw9b" podStartSLOduration=8.950253636 podStartE2EDuration="12.200905764s" totalImagesPullingTime="3.250652128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:21.200082891 +0000 UTC m=+268.390412563" watchObservedRunningTime="2026-07-02 13:19:21.200905764 +0000 UTC m=+268.391235436"
Jul 02 13:19:21.219318 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.219258    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-j2xzr" podStartSLOduration=8.801495146 podStartE2EDuration="12.219236526s" totalImagesPullingTime="3.41774138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:21.218031738 +0000 UTC m=+268.408361440" watchObservedRunningTime="2026-07-02 13:19:21.219236526 +0000 UTC m=+268.409566198"
Jul 02 13:19:21.233547 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.233516    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5589b49b66-qfwq7"]
Jul 02 13:19:21.241110 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.241080    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5589b49b66-qfwq7"]
Jul 02 13:19:21.259032 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:21.258975    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-96v2m" podStartSLOduration=10.929282758 podStartE2EDuration="12.25895121s" totalImagesPullingTime="1.235428129s" totalInitContainerRuntime="94.240323ms" isStatefulPod=true podCreationTimestamp="2026-07-02 13:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:21.257599948 +0000 UTC m=+268.447929620" watchObservedRunningTime="2026-07-02 13:19:21.25895121 +0000 UTC m=+268.449280880"
Jul 02 13:19:23.227090 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:23.227047    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3da4fd-3de2-424a-8106-c214c6d41f6a" path="/var/lib/kubelet/pods/cf3da4fd-3de2-424a-8106-c214c6d41f6a/volumes"
Jul 02 13:19:24.659216 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:24.659167    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:24.659729 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:24.659241    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:24.665136 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:24.665107    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:25.200077 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:25.200048    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:19:25.241057 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:25.240998    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9567bfbb-xt57w"]
Jul 02 13:19:33.993017 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:33.992979    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:33.993482 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:33.993068    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:36.746319 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:19:36.746226    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dl649" podUID="fc8cbf88-1d0a-4f50-afb3-2b828a2902f8"
Jul 02 13:19:37.240993 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:37.240963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dl649"
Jul 02 13:19:40.055050 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:40.055006    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:19:40.057404 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:40.057383    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc8cbf88-1d0a-4f50-afb3-2b828a2902f8-metrics-tls\") pod \"dns-default-dl649\" (UID: \"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8\") " pod="openshift-dns/dns-default-dl649"
Jul 02 13:19:40.244866 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:40.244835    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d2c87\""
Jul 02 13:19:40.252384 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:40.252351    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dl649"
Jul 02 13:19:40.559257 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:40.559225    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dl649"]
Jul 02 13:19:40.563152 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:19:40.563119    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8cbf88_1d0a_4f50_afb3_2b828a2902f8.slice/crio-ef438ea86b165b3dd8785d5e868c30adbc33756a3ebbec69cc3efd63f7e72660 WatchSource:0}: Error finding container ef438ea86b165b3dd8785d5e868c30adbc33756a3ebbec69cc3efd63f7e72660: Status 404 returned error can't find the container with id ef438ea86b165b3dd8785d5e868c30adbc33756a3ebbec69cc3efd63f7e72660
Jul 02 13:19:41.256501 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:41.256465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dl649" event={"ID":"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8","Type":"ContainerStarted","Data":"ef438ea86b165b3dd8785d5e868c30adbc33756a3ebbec69cc3efd63f7e72660"}
Jul 02 13:19:43.267593 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:43.267544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dl649" event={"ID":"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8","Type":"ContainerStarted","Data":"cf6d84b931fb45ae7432f045719ce67e9c8a08dd026e671a813811b9856c3ea5"}
Jul 02 13:19:43.268066 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:43.267617    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dl649" event={"ID":"fc8cbf88-1d0a-4f50-afb3-2b828a2902f8","Type":"ContainerStarted","Data":"24f6dce7664aee1118adf55dd7a28f8e87f39d656dc18a7fadf17e6ce2ea9675"}
Jul 02 13:19:43.268066 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:43.267642    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dl649"
Jul 02 13:19:43.285741 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:43.285677    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dl649" podStartSLOduration=251.389576247 podStartE2EDuration="4m13.285663465s" totalImagesPullingTime="1.896087218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:19:43.283460856 +0000 UTC m=+290.473790546" watchObservedRunningTime="2026-07-02 13:19:43.285663465 +0000 UTC m=+290.475993135"
Jul 02 13:19:46.280693 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:46.280656    2555 generic.go:356] "Generic (PLEG): container finished" podID="17c14389-121d-446e-b6a5-dd612b23c463" containerID="297421307bf61c40781660392c628a8508db4bff8d2d88caf45f50ba6c41271e" exitCode=0
Jul 02 13:19:46.281136 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:46.280731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw" event={"ID":"17c14389-121d-446e-b6a5-dd612b23c463","Type":"ContainerDied","Data":"297421307bf61c40781660392c628a8508db4bff8d2d88caf45f50ba6c41271e"}
Jul 02 13:19:46.281136 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:46.281095    2555 scope.go:117] "RemoveContainer" containerID="297421307bf61c40781660392c628a8508db4bff8d2d88caf45f50ba6c41271e"
Jul 02 13:19:47.286562 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:47.286522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-vszvw" event={"ID":"17c14389-121d-446e-b6a5-dd612b23c463","Type":"ContainerStarted","Data":"7d69acc0ca106c679baca1586bdef8ca0415d552c291db2a0d7402192e7f9fd1"}
Jul 02 13:19:50.271762 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.271714    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7c9567bfbb-xt57w" podUID="fda23ecb-3925-4907-83cf-ffa5350c85b5" containerName="console" containerID="cri-o://157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0" gracePeriod=15
Jul 02 13:19:50.546946 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.546864    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:50.646829 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.646796    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-service-ca\") pod \"fda23ecb-3925-4907-83cf-ffa5350c85b5\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") "
Jul 02 13:19:50.647003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.646846    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m98cq\" (UniqueName: \"kubernetes.io/projected/fda23ecb-3925-4907-83cf-ffa5350c85b5-kube-api-access-m98cq\") pod \"fda23ecb-3925-4907-83cf-ffa5350c85b5\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") "
Jul 02 13:19:50.647003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.646867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-serving-cert\") pod \"fda23ecb-3925-4907-83cf-ffa5350c85b5\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") "
Jul 02 13:19:50.647003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.646907    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-oauth-serving-cert\") pod \"fda23ecb-3925-4907-83cf-ffa5350c85b5\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") "
Jul 02 13:19:50.647003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.646927    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-config\") pod \"fda23ecb-3925-4907-83cf-ffa5350c85b5\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") "
Jul 02 13:19:50.647003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.646957    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-oauth-config\") pod \"fda23ecb-3925-4907-83cf-ffa5350c85b5\" (UID: \"fda23ecb-3925-4907-83cf-ffa5350c85b5\") "
Jul 02 13:19:50.647370 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.647337    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-service-ca" (OuterVolumeSpecName: "service-ca") pod "fda23ecb-3925-4907-83cf-ffa5350c85b5" (UID: "fda23ecb-3925-4907-83cf-ffa5350c85b5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:19:50.647370 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.647362    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fda23ecb-3925-4907-83cf-ffa5350c85b5" (UID: "fda23ecb-3925-4907-83cf-ffa5350c85b5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:19:50.647509 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.647379    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-config" (OuterVolumeSpecName: "console-config") pod "fda23ecb-3925-4907-83cf-ffa5350c85b5" (UID: "fda23ecb-3925-4907-83cf-ffa5350c85b5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:19:50.649291 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.649258    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fda23ecb-3925-4907-83cf-ffa5350c85b5" (UID: "fda23ecb-3925-4907-83cf-ffa5350c85b5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:19:50.649407 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.649321    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fda23ecb-3925-4907-83cf-ffa5350c85b5" (UID: "fda23ecb-3925-4907-83cf-ffa5350c85b5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:19:50.649407 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.649323    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda23ecb-3925-4907-83cf-ffa5350c85b5-kube-api-access-m98cq" (OuterVolumeSpecName: "kube-api-access-m98cq") pod "fda23ecb-3925-4907-83cf-ffa5350c85b5" (UID: "fda23ecb-3925-4907-83cf-ffa5350c85b5"). InnerVolumeSpecName "kube-api-access-m98cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:19:50.747968 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.747940    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:50.747968 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.747964    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m98cq\" (UniqueName: \"kubernetes.io/projected/fda23ecb-3925-4907-83cf-ffa5350c85b5-kube-api-access-m98cq\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:50.747968 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.747974    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:50.748180 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.747984    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:50.748180 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.747993    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:50.748180 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:50.748001    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fda23ecb-3925-4907-83cf-ffa5350c85b5-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:19:51.302831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.302798    2555 generic.go:356] "Generic (PLEG): container finished" podID="fda23ecb-3925-4907-83cf-ffa5350c85b5" containerID="157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0" exitCode=0
Jul 02 13:19:51.303243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.302860    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9567bfbb-xt57w"
Jul 02 13:19:51.303243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.302864    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9567bfbb-xt57w" event={"ID":"fda23ecb-3925-4907-83cf-ffa5350c85b5","Type":"ContainerDied","Data":"157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0"}
Jul 02 13:19:51.303243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.302903    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9567bfbb-xt57w" event={"ID":"fda23ecb-3925-4907-83cf-ffa5350c85b5","Type":"ContainerDied","Data":"60de5d38dbdfa3d94d9dfc701f94d1b22f6856c2273888edb23b21d55c455bbf"}
Jul 02 13:19:51.303243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.302922    2555 scope.go:117] "RemoveContainer" containerID="157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0"
Jul 02 13:19:51.311228 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.311198    2555 scope.go:117] "RemoveContainer" containerID="157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0"
Jul 02 13:19:51.311490 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:19:51.311472    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0\": container with ID starting with 157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0 not found: ID does not exist" containerID="157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0"
Jul 02 13:19:51.311556 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.311498    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0"} err="failed to get container status \"157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0\": rpc error: code = NotFound desc = could not find container \"157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0\": container with ID starting with 157d73c05eb5198fa5077269846ff9c5fa362bd1503af68ea79649ae01f3b6f0 not found: ID does not exist"
Jul 02 13:19:51.319580 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.319525    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9567bfbb-xt57w"]
Jul 02 13:19:51.325175 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:51.325153    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9567bfbb-xt57w"]
Jul 02 13:19:53.150841 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.150813    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:19:53.154111 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.154086    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:19:53.155718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.155681    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:19:53.159494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.159474    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:19:53.164173 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.164154    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 13:19:53.227772 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.227582    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda23ecb-3925-4907-83cf-ffa5350c85b5" path="/var/lib/kubelet/pods/fda23ecb-3925-4907-83cf-ffa5350c85b5/volumes"
Jul 02 13:19:53.273804 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.273776    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dl649"
Jul 02 13:19:53.998063 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:53.998031    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:19:54.001728 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:19:54.001696    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d6d8c96cf-zl9xk"
Jul 02 13:20:37.907967 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.907925    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cf569ddd7-z494n"]
Jul 02 13:20:37.908476 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.908179    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3da4fd-3de2-424a-8106-c214c6d41f6a" containerName="registry"
Jul 02 13:20:37.908476 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.908190    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3da4fd-3de2-424a-8106-c214c6d41f6a" containerName="registry"
Jul 02 13:20:37.908476 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.908205    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda23ecb-3925-4907-83cf-ffa5350c85b5" containerName="console"
Jul 02 13:20:37.908476 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.908210    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda23ecb-3925-4907-83cf-ffa5350c85b5" containerName="console"
Jul 02 13:20:37.908476 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.908261    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda23ecb-3925-4907-83cf-ffa5350c85b5" containerName="console"
Jul 02 13:20:37.908476 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.908270    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3da4fd-3de2-424a-8106-c214c6d41f6a" containerName="registry"
Jul 02 13:20:37.911444 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.911414    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:37.922936 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:37.922901    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf569ddd7-z494n"]
Jul 02 13:20:38.034127 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.034101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-service-ca\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.034300 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.034131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnx4\" (UniqueName: \"kubernetes.io/projected/2af45a21-efca-40f2-beed-f7f0100a582e-kube-api-access-9gnx4\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.034300 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.034165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-serving-cert\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.034300 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.034215    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-oauth-config\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.034300 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.034272    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-console-config\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.034451 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.034307    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-oauth-serving-cert\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.034451 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.034346    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-trusted-ca-bundle\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.135226 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.135199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-serving-cert\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.135226 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.135227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-oauth-config\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.135414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.135252    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-console-config\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.135414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.135276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-oauth-serving-cert\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.135414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.135299    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-trusted-ca-bundle\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.135414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.135346    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-service-ca\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.135414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.135369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnx4\" (UniqueName: \"kubernetes.io/projected/2af45a21-efca-40f2-beed-f7f0100a582e-kube-api-access-9gnx4\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.136129 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.136101    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-oauth-serving-cert\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.136256 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.136101    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-service-ca\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.136256 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.136172    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-console-config\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.136379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.136354    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-trusted-ca-bundle\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.137755 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.137729    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-serving-cert\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.137868 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.137773    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-oauth-config\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.142819 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.142798    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnx4\" (UniqueName: \"kubernetes.io/projected/2af45a21-efca-40f2-beed-f7f0100a582e-kube-api-access-9gnx4\") pod \"console-cf569ddd7-z494n\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") " pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.220953 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.220905    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:38.349028 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.349007    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf569ddd7-z494n"]
Jul 02 13:20:38.351275 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:20:38.351250    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af45a21_efca_40f2_beed_f7f0100a582e.slice/crio-65ab4641955c8be58d7ecbf75003476c3b8f57d636df3b7a4b0540488289918b WatchSource:0}: Error finding container 65ab4641955c8be58d7ecbf75003476c3b8f57d636df3b7a4b0540488289918b: Status 404 returned error can't find the container with id 65ab4641955c8be58d7ecbf75003476c3b8f57d636df3b7a4b0540488289918b
Jul 02 13:20:38.473193 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.473119    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf569ddd7-z494n" event={"ID":"2af45a21-efca-40f2-beed-f7f0100a582e","Type":"ContainerStarted","Data":"e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3"}
Jul 02 13:20:38.473338 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.473196    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf569ddd7-z494n" event={"ID":"2af45a21-efca-40f2-beed-f7f0100a582e","Type":"ContainerStarted","Data":"65ab4641955c8be58d7ecbf75003476c3b8f57d636df3b7a4b0540488289918b"}
Jul 02 13:20:38.490846 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:38.490798    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cf569ddd7-z494n" podStartSLOduration=1.490783805 podStartE2EDuration="1.490783805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:20:38.489313886 +0000 UTC m=+345.679643563" watchObservedRunningTime="2026-07-02 13:20:38.490783805 +0000 UTC m=+345.681113476"
Jul 02 13:20:47.266284 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.266197    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf569ddd7-z494n"]
Jul 02 13:20:47.297332 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.297308    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58bf85db86-mqlrh"]
Jul 02 13:20:47.299310 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.299293    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.308887 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.308865    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58bf85db86-mqlrh"]
Jul 02 13:20:47.399003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.398972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-service-ca\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.399003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.399003    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-oauth-serving-cert\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.399194 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.399031    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-trusted-ca-bundle\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.399194 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.399053    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-serving-cert\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.399194 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.399089    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-oauth-config\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.399194 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.399165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-console-config\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.399194 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.399181    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tl9h\" (UniqueName: \"kubernetes.io/projected/2549556c-5edb-41a6-848d-7e49da4dc436-kube-api-access-4tl9h\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.500268 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.500238    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-serving-cert\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.500364 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.500289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-oauth-config\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.500364 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.500342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-console-config\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.500462 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.500368    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tl9h\" (UniqueName: \"kubernetes.io/projected/2549556c-5edb-41a6-848d-7e49da4dc436-kube-api-access-4tl9h\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.500462 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.500430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-service-ca\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.500576 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.500556    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-oauth-serving-cert\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.500633 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.500614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-trusted-ca-bundle\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.501030 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.501006    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-console-config\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.501379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.501353    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-trusted-ca-bundle\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.501450 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.501380    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-service-ca\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.501450 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.501363    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-oauth-serving-cert\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.502766 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.502747    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-serving-cert\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.502857 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.502794    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-oauth-config\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.507798 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.507774    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tl9h\" (UniqueName: \"kubernetes.io/projected/2549556c-5edb-41a6-848d-7e49da4dc436-kube-api-access-4tl9h\") pod \"console-58bf85db86-mqlrh\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") " pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.609061 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.609038    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:47.731356 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:47.731333    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58bf85db86-mqlrh"]
Jul 02 13:20:47.733884 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:20:47.733859    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2549556c_5edb_41a6_848d_7e49da4dc436.slice/crio-f72c379221a270a6291011f2fb80c9248ced6c6066946946f53f90ad9b07280d WatchSource:0}: Error finding container f72c379221a270a6291011f2fb80c9248ced6c6066946946f53f90ad9b07280d: Status 404 returned error can't find the container with id f72c379221a270a6291011f2fb80c9248ced6c6066946946f53f90ad9b07280d
Jul 02 13:20:48.221067 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:48.221040    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:20:48.510646 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:48.510555    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58bf85db86-mqlrh" event={"ID":"2549556c-5edb-41a6-848d-7e49da4dc436","Type":"ContainerStarted","Data":"8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff"}
Jul 02 13:20:48.510646 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:48.510612    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58bf85db86-mqlrh" event={"ID":"2549556c-5edb-41a6-848d-7e49da4dc436","Type":"ContainerStarted","Data":"f72c379221a270a6291011f2fb80c9248ced6c6066946946f53f90ad9b07280d"}
Jul 02 13:20:48.526751 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:48.526696    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58bf85db86-mqlrh" podStartSLOduration=1.526682615 podStartE2EDuration="1.526682615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:20:48.526272208 +0000 UTC m=+355.716601905" watchObservedRunningTime="2026-07-02 13:20:48.526682615 +0000 UTC m=+355.717012287"
Jul 02 13:20:57.609936 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:57.609905    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:57.610408 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:57.609999    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:57.614503 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:57.614483    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:58.548880 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:58.548856    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:20:58.593619 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:20:58.593586    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744f7db998-t6tr9"]
Jul 02 13:21:12.286875 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.286816    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-cf569ddd7-z494n" podUID="2af45a21-efca-40f2-beed-f7f0100a582e" containerName="console" containerID="cri-o://e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3" gracePeriod=15
Jul 02 13:21:12.520203 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.520181    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:21:12.593753 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.593728    2555 generic.go:356] "Generic (PLEG): container finished" podID="2af45a21-efca-40f2-beed-f7f0100a582e" containerID="e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3" exitCode=0
Jul 02 13:21:12.593886 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.593771    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf569ddd7-z494n"
Jul 02 13:21:12.593886 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.593774    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf569ddd7-z494n" event={"ID":"2af45a21-efca-40f2-beed-f7f0100a582e","Type":"ContainerDied","Data":"e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3"}
Jul 02 13:21:12.593964 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.593907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf569ddd7-z494n" event={"ID":"2af45a21-efca-40f2-beed-f7f0100a582e","Type":"ContainerDied","Data":"65ab4641955c8be58d7ecbf75003476c3b8f57d636df3b7a4b0540488289918b"}
Jul 02 13:21:12.593964 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.593935    2555 scope.go:117] "RemoveContainer" containerID="e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3"
Jul 02 13:21:12.601545 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.601528    2555 scope.go:117] "RemoveContainer" containerID="e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3"
Jul 02 13:21:12.601808 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:21:12.601790    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3\": container with ID starting with e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3 not found: ID does not exist" containerID="e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3"
Jul 02 13:21:12.601867 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.601821    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3"} err="failed to get container status \"e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3\": rpc error: code = NotFound desc = could not find container \"e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3\": container with ID starting with e6371e6e3a6595f3f5ab836f7a7ee672a1cfa7086aac8587fecfdb8dae5db8c3 not found: ID does not exist"
Jul 02 13:21:12.676489 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676464    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-trusted-ca-bundle\") pod \"2af45a21-efca-40f2-beed-f7f0100a582e\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") "
Jul 02 13:21:12.676578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676523    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-console-config\") pod \"2af45a21-efca-40f2-beed-f7f0100a582e\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") "
Jul 02 13:21:12.676578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676552    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-oauth-config\") pod \"2af45a21-efca-40f2-beed-f7f0100a582e\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") "
Jul 02 13:21:12.676695 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676594    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gnx4\" (UniqueName: \"kubernetes.io/projected/2af45a21-efca-40f2-beed-f7f0100a582e-kube-api-access-9gnx4\") pod \"2af45a21-efca-40f2-beed-f7f0100a582e\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") "
Jul 02 13:21:12.676695 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676650    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-oauth-serving-cert\") pod \"2af45a21-efca-40f2-beed-f7f0100a582e\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") "
Jul 02 13:21:12.676695 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676679    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-service-ca\") pod \"2af45a21-efca-40f2-beed-f7f0100a582e\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") "
Jul 02 13:21:12.676878 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676729    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-serving-cert\") pod \"2af45a21-efca-40f2-beed-f7f0100a582e\" (UID: \"2af45a21-efca-40f2-beed-f7f0100a582e\") "
Jul 02 13:21:12.677002 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676883    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2af45a21-efca-40f2-beed-f7f0100a582e" (UID: "2af45a21-efca-40f2-beed-f7f0100a582e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:12.677120 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.676995    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-console-config" (OuterVolumeSpecName: "console-config") pod "2af45a21-efca-40f2-beed-f7f0100a582e" (UID: "2af45a21-efca-40f2-beed-f7f0100a582e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:12.677120 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.677051    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2af45a21-efca-40f2-beed-f7f0100a582e" (UID: "2af45a21-efca-40f2-beed-f7f0100a582e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:12.677120 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.677062    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-service-ca" (OuterVolumeSpecName: "service-ca") pod "2af45a21-efca-40f2-beed-f7f0100a582e" (UID: "2af45a21-efca-40f2-beed-f7f0100a582e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:12.678574 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.678547    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2af45a21-efca-40f2-beed-f7f0100a582e" (UID: "2af45a21-efca-40f2-beed-f7f0100a582e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:21:12.678809 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.678776    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af45a21-efca-40f2-beed-f7f0100a582e-kube-api-access-9gnx4" (OuterVolumeSpecName: "kube-api-access-9gnx4") pod "2af45a21-efca-40f2-beed-f7f0100a582e" (UID: "2af45a21-efca-40f2-beed-f7f0100a582e"). InnerVolumeSpecName "kube-api-access-9gnx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:21:12.678809 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.678793    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2af45a21-efca-40f2-beed-f7f0100a582e" (UID: "2af45a21-efca-40f2-beed-f7f0100a582e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:21:12.777399 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.777375    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:12.777399 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.777397    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:12.777511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.777406    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:12.777511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.777415    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:12.777511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.777424    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2af45a21-efca-40f2-beed-f7f0100a582e-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:12.777511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.777432    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gnx4\" (UniqueName: \"kubernetes.io/projected/2af45a21-efca-40f2-beed-f7f0100a582e-kube-api-access-9gnx4\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:12.777511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.777441    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2af45a21-efca-40f2-beed-f7f0100a582e-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:12.913880 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.913858    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf569ddd7-z494n"]
Jul 02 13:21:12.917250 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:12.917225    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cf569ddd7-z494n"]
Jul 02 13:21:13.224553 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:13.224476    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af45a21-efca-40f2-beed-f7f0100a582e" path="/var/lib/kubelet/pods/2af45a21-efca-40f2-beed-f7f0100a582e/volumes"
Jul 02 13:21:23.618381 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.618321    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-744f7db998-t6tr9" podUID="5f0016e2-2afa-4dba-94af-46dcc6aff297" containerName="console" containerID="cri-o://e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0" gracePeriod=15
Jul 02 13:21:23.861547 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.861519    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:21:23.968827 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.968729    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-oauth-serving-cert\") pod \"5f0016e2-2afa-4dba-94af-46dcc6aff297\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") "
Jul 02 13:21:23.968827 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.968802    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-oauth-config\") pod \"5f0016e2-2afa-4dba-94af-46dcc6aff297\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") "
Jul 02 13:21:23.969065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.968843    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-service-ca\") pod \"5f0016e2-2afa-4dba-94af-46dcc6aff297\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") "
Jul 02 13:21:23.969065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.968879    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-trusted-ca-bundle\") pod \"5f0016e2-2afa-4dba-94af-46dcc6aff297\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") "
Jul 02 13:21:23.969065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.968950    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-config\") pod \"5f0016e2-2afa-4dba-94af-46dcc6aff297\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") "
Jul 02 13:21:23.969065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.969008    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-serving-cert\") pod \"5f0016e2-2afa-4dba-94af-46dcc6aff297\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") "
Jul 02 13:21:23.969065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.969047    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnfbt\" (UniqueName: \"kubernetes.io/projected/5f0016e2-2afa-4dba-94af-46dcc6aff297-kube-api-access-xnfbt\") pod \"5f0016e2-2afa-4dba-94af-46dcc6aff297\" (UID: \"5f0016e2-2afa-4dba-94af-46dcc6aff297\") "
Jul 02 13:21:23.969326 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.969251    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5f0016e2-2afa-4dba-94af-46dcc6aff297" (UID: "5f0016e2-2afa-4dba-94af-46dcc6aff297"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:23.969390 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.969329    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5f0016e2-2afa-4dba-94af-46dcc6aff297" (UID: "5f0016e2-2afa-4dba-94af-46dcc6aff297"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:23.969447 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.969427    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-config" (OuterVolumeSpecName: "console-config") pod "5f0016e2-2afa-4dba-94af-46dcc6aff297" (UID: "5f0016e2-2afa-4dba-94af-46dcc6aff297"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:23.969667 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.969641    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-service-ca" (OuterVolumeSpecName: "service-ca") pod "5f0016e2-2afa-4dba-94af-46dcc6aff297" (UID: "5f0016e2-2afa-4dba-94af-46dcc6aff297"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:21:23.971064 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.971041    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5f0016e2-2afa-4dba-94af-46dcc6aff297" (UID: "5f0016e2-2afa-4dba-94af-46dcc6aff297"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:21:23.971537 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.971509    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5f0016e2-2afa-4dba-94af-46dcc6aff297" (UID: "5f0016e2-2afa-4dba-94af-46dcc6aff297"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:21:23.971628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:23.971536    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0016e2-2afa-4dba-94af-46dcc6aff297-kube-api-access-xnfbt" (OuterVolumeSpecName: "kube-api-access-xnfbt") pod "5f0016e2-2afa-4dba-94af-46dcc6aff297" (UID: "5f0016e2-2afa-4dba-94af-46dcc6aff297"). InnerVolumeSpecName "kube-api-access-xnfbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:21:24.070628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.070563    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfbt\" (UniqueName: \"kubernetes.io/projected/5f0016e2-2afa-4dba-94af-46dcc6aff297-kube-api-access-xnfbt\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:24.070628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.070615    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:24.070628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.070625    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:24.070628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.070637    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:24.070628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.070646    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:24.070963 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.070653    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:24.070963 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.070662    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f0016e2-2afa-4dba-94af-46dcc6aff297-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:21:24.640588 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.640552    2555 generic.go:356] "Generic (PLEG): container finished" podID="5f0016e2-2afa-4dba-94af-46dcc6aff297" containerID="e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0" exitCode=0
Jul 02 13:21:24.641049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.640593    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-744f7db998-t6tr9"
Jul 02 13:21:24.641049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.640630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7db998-t6tr9" event={"ID":"5f0016e2-2afa-4dba-94af-46dcc6aff297","Type":"ContainerDied","Data":"e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0"}
Jul 02 13:21:24.641049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.640677    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-744f7db998-t6tr9" event={"ID":"5f0016e2-2afa-4dba-94af-46dcc6aff297","Type":"ContainerDied","Data":"6a20f3eae0ce9851ff5e5bcae4d04cac6561d37a4c156f7fd455dc3d8aac4c02"}
Jul 02 13:21:24.641049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.640697    2555 scope.go:117] "RemoveContainer" containerID="e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0"
Jul 02 13:21:24.652155 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.652141    2555 scope.go:117] "RemoveContainer" containerID="e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0"
Jul 02 13:21:24.652417 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:21:24.652401    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0\": container with ID starting with e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0 not found: ID does not exist" containerID="e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0"
Jul 02 13:21:24.652460 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.652425    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0"} err="failed to get container status \"e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0\": rpc error: code = NotFound desc = could not find container \"e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0\": container with ID starting with e67e82adac03694b74170ca67b632a4727e7968ad80ba356b240a3c64ace20e0 not found: ID does not exist"
Jul 02 13:21:24.664397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.664369    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-744f7db998-t6tr9"]
Jul 02 13:21:24.668167 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:24.668142    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-744f7db998-t6tr9"]
Jul 02 13:21:25.224858 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:21:25.224817    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0016e2-2afa-4dba-94af-46dcc6aff297" path="/var/lib/kubelet/pods/5f0016e2-2afa-4dba-94af-46dcc6aff297/volumes"
Jul 02 13:23:05.451206 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.451172    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"]
Jul 02 13:23:05.451601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.451409    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0016e2-2afa-4dba-94af-46dcc6aff297" containerName="console"
Jul 02 13:23:05.451601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.451419    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0016e2-2afa-4dba-94af-46dcc6aff297" containerName="console"
Jul 02 13:23:05.451601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.451435    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af45a21-efca-40f2-beed-f7f0100a582e" containerName="console"
Jul 02 13:23:05.451601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.451440    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af45a21-efca-40f2-beed-f7f0100a582e" containerName="console"
Jul 02 13:23:05.451601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.451488    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0016e2-2afa-4dba-94af-46dcc6aff297" containerName="console"
Jul 02 13:23:05.451601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.451496    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af45a21-efca-40f2-beed-f7f0100a582e" containerName="console"
Jul 02 13:23:05.453344 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.453329    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.455893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.455871    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 02 13:23:05.457178 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.457159    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 02 13:23:05.457274 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.457159    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lr9qr\""
Jul 02 13:23:05.468121 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.468102    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"]
Jul 02 13:23:05.521646 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.521623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbm8r\" (UniqueName: \"kubernetes.io/projected/70f4d383-d5cf-47e3-9273-44b1bc357b49-kube-api-access-jbm8r\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.521786 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.521685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.521786 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.521736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.622907 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.622885    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.623001 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.622914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.623001 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.622952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbm8r\" (UniqueName: \"kubernetes.io/projected/70f4d383-d5cf-47e3-9273-44b1bc357b49-kube-api-access-jbm8r\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.623242 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.623226    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.623278 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.623260    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.631258 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.631234    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbm8r\" (UniqueName: \"kubernetes.io/projected/70f4d383-d5cf-47e3-9273-44b1bc357b49-kube-api-access-jbm8r\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.761831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.761778    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:05.883507 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.883481    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"]
Jul 02 13:23:05.885928 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:23:05.885903    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f4d383_d5cf_47e3_9273_44b1bc357b49.slice/crio-41c01c61401a742ab6253b78f13544188928f9c3085e2128ec18a9ea6012e5e5 WatchSource:0}: Error finding container 41c01c61401a742ab6253b78f13544188928f9c3085e2128ec18a9ea6012e5e5: Status 404 returned error can't find the container with id 41c01c61401a742ab6253b78f13544188928f9c3085e2128ec18a9ea6012e5e5
Jul 02 13:23:05.887747 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.887731    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:23:05.998242 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:05.998214    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k" event={"ID":"70f4d383-d5cf-47e3-9273-44b1bc357b49","Type":"ContainerStarted","Data":"41c01c61401a742ab6253b78f13544188928f9c3085e2128ec18a9ea6012e5e5"}
Jul 02 13:23:11.019759 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:11.019721    2555 generic.go:356] "Generic (PLEG): container finished" podID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerID="e44f588789a998d95b720ae5426d305853ac725ba3131e51c6a9df11a7068c5f" exitCode=0
Jul 02 13:23:11.020199 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:11.019799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k" event={"ID":"70f4d383-d5cf-47e3-9273-44b1bc357b49","Type":"ContainerDied","Data":"e44f588789a998d95b720ae5426d305853ac725ba3131e51c6a9df11a7068c5f"}
Jul 02 13:23:13.029255 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:13.029225    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k" event={"ID":"70f4d383-d5cf-47e3-9273-44b1bc357b49","Type":"ContainerStarted","Data":"70da64a60f75f5f1c02483f2c98061c7e5f66089f3f8ecedf189e8cf8a954902"}
Jul 02 13:23:14.034243 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:14.034207    2555 generic.go:356] "Generic (PLEG): container finished" podID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerID="70da64a60f75f5f1c02483f2c98061c7e5f66089f3f8ecedf189e8cf8a954902" exitCode=0
Jul 02 13:23:14.034631 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:14.034292    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k" event={"ID":"70f4d383-d5cf-47e3-9273-44b1bc357b49","Type":"ContainerDied","Data":"70da64a60f75f5f1c02483f2c98061c7e5f66089f3f8ecedf189e8cf8a954902"}
Jul 02 13:23:22.068002 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:22.067965    2555 generic.go:356] "Generic (PLEG): container finished" podID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerID="1002bdb7590d55ea975828de2e2f636456aca8547a153deb83a60ceb29794bbc" exitCode=0
Jul 02 13:23:22.068374 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:22.068039    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k" event={"ID":"70f4d383-d5cf-47e3-9273-44b1bc357b49","Type":"ContainerDied","Data":"1002bdb7590d55ea975828de2e2f636456aca8547a153deb83a60ceb29794bbc"}
Jul 02 13:23:23.188344 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.188323    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:23.268387 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.268359    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-util\") pod \"70f4d383-d5cf-47e3-9273-44b1bc357b49\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") "
Jul 02 13:23:23.268529 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.268394    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-bundle\") pod \"70f4d383-d5cf-47e3-9273-44b1bc357b49\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") "
Jul 02 13:23:23.268529 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.268474    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbm8r\" (UniqueName: \"kubernetes.io/projected/70f4d383-d5cf-47e3-9273-44b1bc357b49-kube-api-access-jbm8r\") pod \"70f4d383-d5cf-47e3-9273-44b1bc357b49\" (UID: \"70f4d383-d5cf-47e3-9273-44b1bc357b49\") "
Jul 02 13:23:23.269111 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.269088    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-bundle" (OuterVolumeSpecName: "bundle") pod "70f4d383-d5cf-47e3-9273-44b1bc357b49" (UID: "70f4d383-d5cf-47e3-9273-44b1bc357b49"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:23:23.270611 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.270585    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f4d383-d5cf-47e3-9273-44b1bc357b49-kube-api-access-jbm8r" (OuterVolumeSpecName: "kube-api-access-jbm8r") pod "70f4d383-d5cf-47e3-9273-44b1bc357b49" (UID: "70f4d383-d5cf-47e3-9273-44b1bc357b49"). InnerVolumeSpecName "kube-api-access-jbm8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:23:23.272516 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.272490    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-util" (OuterVolumeSpecName: "util") pod "70f4d383-d5cf-47e3-9273-44b1bc357b49" (UID: "70f4d383-d5cf-47e3-9273-44b1bc357b49"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 13:23:23.369308 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.369281    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbm8r\" (UniqueName: \"kubernetes.io/projected/70f4d383-d5cf-47e3-9273-44b1bc357b49-kube-api-access-jbm8r\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:23:23.369308 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.369304    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-util\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:23:23.369308 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:23.369314    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70f4d383-d5cf-47e3-9273-44b1bc357b49-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:23:24.076486 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:24.076461    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k"
Jul 02 13:23:24.076486 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:24.076472    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58972d8k" event={"ID":"70f4d383-d5cf-47e3-9273-44b1bc357b49","Type":"ContainerDied","Data":"41c01c61401a742ab6253b78f13544188928f9c3085e2128ec18a9ea6012e5e5"}
Jul 02 13:23:24.076680 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:24.076513    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41c01c61401a742ab6253b78f13544188928f9c3085e2128ec18a9ea6012e5e5"
Jul 02 13:23:27.686493 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.686460    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"]
Jul 02 13:23:27.686987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.686870    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerName="pull"
Jul 02 13:23:27.686987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.686887    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerName="pull"
Jul 02 13:23:27.686987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.686900    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerName="extract"
Jul 02 13:23:27.686987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.686908    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerName="extract"
Jul 02 13:23:27.686987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.686930    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerName="util"
Jul 02 13:23:27.686987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.686967    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerName="util"
Jul 02 13:23:27.687285 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.687043    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f4d383-d5cf-47e3-9273-44b1bc357b49" containerName="extract"
Jul 02 13:23:27.731321 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.731297    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"]
Jul 02 13:23:27.731447 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.731390    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:27.736805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.736698    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-bxrbd\""
Jul 02 13:23:27.736805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.736792    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 13:23:27.737174 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.737156    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 13:23:27.737664 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.737646    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 13:23:27.798572 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.798547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d80fb2cf-8bb1-4584-9060-e85f772d8e8e-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl\" (UID: \"d80fb2cf-8bb1-4584-9060-e85f772d8e8e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:27.798679 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.798610    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdsg\" (UniqueName: \"kubernetes.io/projected/d80fb2cf-8bb1-4584-9060-e85f772d8e8e-kube-api-access-mrdsg\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl\" (UID: \"d80fb2cf-8bb1-4584-9060-e85f772d8e8e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:27.899293 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.899272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d80fb2cf-8bb1-4584-9060-e85f772d8e8e-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl\" (UID: \"d80fb2cf-8bb1-4584-9060-e85f772d8e8e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:27.899391 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.899320    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdsg\" (UniqueName: \"kubernetes.io/projected/d80fb2cf-8bb1-4584-9060-e85f772d8e8e-kube-api-access-mrdsg\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl\" (UID: \"d80fb2cf-8bb1-4584-9060-e85f772d8e8e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:27.901462 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.901437    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d80fb2cf-8bb1-4584-9060-e85f772d8e8e-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl\" (UID: \"d80fb2cf-8bb1-4584-9060-e85f772d8e8e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:27.914070 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:27.914051    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdsg\" (UniqueName: \"kubernetes.io/projected/d80fb2cf-8bb1-4584-9060-e85f772d8e8e-kube-api-access-mrdsg\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl\" (UID: \"d80fb2cf-8bb1-4584-9060-e85f772d8e8e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:28.041514 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:28.041466    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:28.187006 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:28.186930    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"]
Jul 02 13:23:28.190209 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:23:28.190183    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80fb2cf_8bb1_4584_9060_e85f772d8e8e.slice/crio-c5339fdc86afab4cdf88c580867d565d4ea9bbd44d6da53880acd006a39c5f5d WatchSource:0}: Error finding container c5339fdc86afab4cdf88c580867d565d4ea9bbd44d6da53880acd006a39c5f5d: Status 404 returned error can't find the container with id c5339fdc86afab4cdf88c580867d565d4ea9bbd44d6da53880acd006a39c5f5d
Jul 02 13:23:29.099298 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:29.099265    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl" event={"ID":"d80fb2cf-8bb1-4584-9060-e85f772d8e8e","Type":"ContainerStarted","Data":"c5339fdc86afab4cdf88c580867d565d4ea9bbd44d6da53880acd006a39c5f5d"}
Jul 02 13:23:33.117637 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.117603    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl" event={"ID":"d80fb2cf-8bb1-4584-9060-e85f772d8e8e","Type":"ContainerStarted","Data":"9d2dfb65d7346bffeae180810dc10db89cb86e30dda5d280b01250ec96bd3270"}
Jul 02 13:23:33.118093 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.117737    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:33.137740 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.137680    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl" podStartSLOduration=1.659317645 podStartE2EDuration="6.137665568s" totalImagesPullingTime="4.478347923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:33.135747818 +0000 UTC m=+520.326077489" watchObservedRunningTime="2026-07-02 13:23:33.137665568 +0000 UTC m=+520.327995239"
Jul 02 13:23:33.417668 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.417588    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-mx44d"]
Jul 02 13:23:33.433233 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.433172    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-mx44d"]
Jul 02 13:23:33.433366 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.433256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.436107 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.436064    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 13:23:33.436235 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.436139    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xg4zl\""
Jul 02 13:23:33.436235 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.436072    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 02 13:23:33.541602 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.541579    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.541749 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.541609    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eddde3b3-e890-46f5-9b12-db0d9c818678-cabundle0\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.541749 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.541626    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvg28\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-kube-api-access-jvg28\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.642750 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.642722    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.642885 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.642759    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eddde3b3-e890-46f5-9b12-db0d9c818678-cabundle0\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.642885 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.642776    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvg28\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-kube-api-access-jvg28\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.642885 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:33.642878    2555 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 02 13:23:33.642998 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:33.642898    2555 secret.go:281] references non-existent secret key: ca.crt
Jul 02 13:23:33.642998 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:33.642908    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 13:23:33.642998 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:33.642922    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-mx44d: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 13:23:33.642998 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:33.642979    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates podName:eddde3b3-e890-46f5-9b12-db0d9c818678 nodeName:}" failed. No retries permitted until 2026-07-02 13:23:34.142960323 +0000 UTC m=+521.333290003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates") pod "keda-operator-68bff59c-mx44d" (UID: "eddde3b3-e890-46f5-9b12-db0d9c818678") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 02 13:23:33.643402 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.643377    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eddde3b3-e890-46f5-9b12-db0d9c818678-cabundle0\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:33.652449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:33.652425    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvg28\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-kube-api-access-jvg28\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:34.147397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:34.147365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:34.147869 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:34.147516    2555 secret.go:281] references non-existent secret key: ca.crt
Jul 02 13:23:34.147869 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:34.147537    2555 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 13:23:34.147869 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:34.147549    2555 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-mx44d: references non-existent secret key: ca.crt
Jul 02 13:23:34.147869 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:23:34.147604    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates podName:eddde3b3-e890-46f5-9b12-db0d9c818678 nodeName:}" failed. No retries permitted until 2026-07-02 13:23:35.147587138 +0000 UTC m=+522.337916798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates") pod "keda-operator-68bff59c-mx44d" (UID: "eddde3b3-e890-46f5-9b12-db0d9c818678") : references non-existent secret key: ca.crt
Jul 02 13:23:35.154814 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:35.154783    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:35.157052 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:35.157034    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eddde3b3-e890-46f5-9b12-db0d9c818678-certificates\") pod \"keda-operator-68bff59c-mx44d\" (UID: \"eddde3b3-e890-46f5-9b12-db0d9c818678\") " pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:35.244342 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:35.244312    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:35.363425 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:35.363392    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-mx44d"]
Jul 02 13:23:35.367040 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:23:35.367014    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddde3b3_e890_46f5_9b12_db0d9c818678.slice/crio-ad0fe0a84b14f6affbc92206e9d0bfea30ec7e5b18a89ae96cbdaa0dadf07be7 WatchSource:0}: Error finding container ad0fe0a84b14f6affbc92206e9d0bfea30ec7e5b18a89ae96cbdaa0dadf07be7: Status 404 returned error can't find the container with id ad0fe0a84b14f6affbc92206e9d0bfea30ec7e5b18a89ae96cbdaa0dadf07be7
Jul 02 13:23:36.130952 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:36.130919    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-mx44d" event={"ID":"eddde3b3-e890-46f5-9b12-db0d9c818678","Type":"ContainerStarted","Data":"ad0fe0a84b14f6affbc92206e9d0bfea30ec7e5b18a89ae96cbdaa0dadf07be7"}
Jul 02 13:23:54.124022 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:54.123945    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-bbqxl"
Jul 02 13:23:59.229160 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:59.229124    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-mx44d" event={"ID":"eddde3b3-e890-46f5-9b12-db0d9c818678","Type":"ContainerStarted","Data":"9dc36c3b605b6002cf6409f8c1f19ba9b88f933775e1ff8bd9e9cb93373b5373"}
Jul 02 13:23:59.229578 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:59.229254    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:23:59.245773 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:23:59.245726    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-mx44d" podStartSLOduration=3.09233843 podStartE2EDuration="26.245697603s" totalImagesPullingTime="23.153359173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:23:59.244120365 +0000 UTC m=+546.434450036" watchObservedRunningTime="2026-07-02 13:23:59.245697603 +0000 UTC m=+546.436027272"
Jul 02 13:24:20.234666 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:20.234637    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-mx44d"
Jul 02 13:24:44.246928 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.246895    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-jmppc"]
Jul 02 13:24:44.251045 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.251025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:44.253634 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.253605    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 13:24:44.253634 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.253623    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"]
Jul 02 13:24:44.253831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.253733    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 13:24:44.253831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.253756    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 13:24:44.253831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.253772    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-2b9b4\""
Jul 02 13:24:44.255873 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.255854    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.258119 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.258101    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 02 13:24:44.258119 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.258124    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-vgfzp\""
Jul 02 13:24:44.260828 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.260808    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-jmppc"]
Jul 02 13:24:44.268081 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.268059    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"]
Jul 02 13:24:44.420947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.420922    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqdq7\" (UniqueName: \"kubernetes.io/projected/e09a3183-0bb6-4744-be2b-ecf1df64fa4c-kube-api-access-vqdq7\") pod \"llmisvc-controller-manager-67d95fdc48-9vkdn\" (UID: \"e09a3183-0bb6-4744-be2b-ecf1df64fa4c\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.421106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.420961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert\") pod \"kserve-controller-manager-694f5b7594-jmppc\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") " pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:44.421106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.421006    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vbg\" (UniqueName: \"kubernetes.io/projected/3638a2f4-0935-4d00-b448-a9ef5e251c03-kube-api-access-t7vbg\") pod \"kserve-controller-manager-694f5b7594-jmppc\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") " pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:44.421106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.421093    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e09a3183-0bb6-4744-be2b-ecf1df64fa4c-cert\") pod \"llmisvc-controller-manager-67d95fdc48-9vkdn\" (UID: \"e09a3183-0bb6-4744-be2b-ecf1df64fa4c\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.521918 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.521849    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqdq7\" (UniqueName: \"kubernetes.io/projected/e09a3183-0bb6-4744-be2b-ecf1df64fa4c-kube-api-access-vqdq7\") pod \"llmisvc-controller-manager-67d95fdc48-9vkdn\" (UID: \"e09a3183-0bb6-4744-be2b-ecf1df64fa4c\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.521918 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.521887    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert\") pod \"kserve-controller-manager-694f5b7594-jmppc\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") " pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:44.521918 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.521917    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vbg\" (UniqueName: \"kubernetes.io/projected/3638a2f4-0935-4d00-b448-a9ef5e251c03-kube-api-access-t7vbg\") pod \"kserve-controller-manager-694f5b7594-jmppc\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") " pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:44.522153 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.521934    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e09a3183-0bb6-4744-be2b-ecf1df64fa4c-cert\") pod \"llmisvc-controller-manager-67d95fdc48-9vkdn\" (UID: \"e09a3183-0bb6-4744-be2b-ecf1df64fa4c\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.522153 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:24:44.522002    2555 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 02 13:24:44.522153 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:24:44.522073    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert podName:3638a2f4-0935-4d00-b448-a9ef5e251c03 nodeName:}" failed. No retries permitted until 2026-07-02 13:24:45.022054029 +0000 UTC m=+592.212383685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert") pod "kserve-controller-manager-694f5b7594-jmppc" (UID: "3638a2f4-0935-4d00-b448-a9ef5e251c03") : secret "kserve-webhook-server-cert" not found
Jul 02 13:24:44.524110 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.524094    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e09a3183-0bb6-4744-be2b-ecf1df64fa4c-cert\") pod \"llmisvc-controller-manager-67d95fdc48-9vkdn\" (UID: \"e09a3183-0bb6-4744-be2b-ecf1df64fa4c\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.530852 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.530832    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqdq7\" (UniqueName: \"kubernetes.io/projected/e09a3183-0bb6-4744-be2b-ecf1df64fa4c-kube-api-access-vqdq7\") pod \"llmisvc-controller-manager-67d95fdc48-9vkdn\" (UID: \"e09a3183-0bb6-4744-be2b-ecf1df64fa4c\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.531055 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.531033    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vbg\" (UniqueName: \"kubernetes.io/projected/3638a2f4-0935-4d00-b448-a9ef5e251c03-kube-api-access-t7vbg\") pod \"kserve-controller-manager-694f5b7594-jmppc\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") " pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:44.569426 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.569402    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:44.715344 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:44.715324    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"]
Jul 02 13:24:44.717300 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:24:44.717274    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode09a3183_0bb6_4744_be2b_ecf1df64fa4c.slice/crio-36d8f1942b8ca487acb34a0f7369f1c3a6e808080669db4acadfea782d8eddc4 WatchSource:0}: Error finding container 36d8f1942b8ca487acb34a0f7369f1c3a6e808080669db4acadfea782d8eddc4: Status 404 returned error can't find the container with id 36d8f1942b8ca487acb34a0f7369f1c3a6e808080669db4acadfea782d8eddc4
Jul 02 13:24:45.026796 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:45.026753    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert\") pod \"kserve-controller-manager-694f5b7594-jmppc\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") " pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:45.029117 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:45.029089    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert\") pod \"kserve-controller-manager-694f5b7594-jmppc\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") " pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:45.164235 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:45.164198    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:45.376055 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:45.376019    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-jmppc"]
Jul 02 13:24:45.402496 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:45.402467    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn" event={"ID":"e09a3183-0bb6-4744-be2b-ecf1df64fa4c","Type":"ContainerStarted","Data":"36d8f1942b8ca487acb34a0f7369f1c3a6e808080669db4acadfea782d8eddc4"}
Jul 02 13:24:45.505125 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:24:45.505081    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3638a2f4_0935_4d00_b448_a9ef5e251c03.slice/crio-3d2e6db637a2769b9f00a4d41280bd28eab0ab21df98b9cd28242c5e63d2fe8f WatchSource:0}: Error finding container 3d2e6db637a2769b9f00a4d41280bd28eab0ab21df98b9cd28242c5e63d2fe8f: Status 404 returned error can't find the container with id 3d2e6db637a2769b9f00a4d41280bd28eab0ab21df98b9cd28242c5e63d2fe8f
Jul 02 13:24:46.411260 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:46.411190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-jmppc" event={"ID":"3638a2f4-0935-4d00-b448-a9ef5e251c03","Type":"ContainerStarted","Data":"3d2e6db637a2769b9f00a4d41280bd28eab0ab21df98b9cd28242c5e63d2fe8f"}
Jul 02 13:24:47.417459 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:47.417419    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn" event={"ID":"e09a3183-0bb6-4744-be2b-ecf1df64fa4c","Type":"ContainerStarted","Data":"6498360b75eff1073efa972f279a42bff91d0f2c856736fb14b2eb50a10d05af"}
Jul 02 13:24:47.417894 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:47.417583    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:24:47.433981 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:47.433936    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn" podStartSLOduration=1.483212409 podStartE2EDuration="3.433919278s" totalImagesPullingTime="1.950706869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:47.431988381 +0000 UTC m=+594.622318043" watchObservedRunningTime="2026-07-02 13:24:47.433919278 +0000 UTC m=+594.624248927"
Jul 02 13:24:48.425257 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:48.425185    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-jmppc" event={"ID":"3638a2f4-0935-4d00-b448-a9ef5e251c03","Type":"ContainerStarted","Data":"7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80"}
Jul 02 13:24:48.425686 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:48.425407    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:24:48.448801 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:48.448756    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-694f5b7594-jmppc" podStartSLOduration=1.7887559240000002 podStartE2EDuration="4.448740751s" totalImagesPullingTime="2.659984827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:24:48.447333426 +0000 UTC m=+595.637663108" watchObservedRunningTime="2026-07-02 13:24:48.448740751 +0000 UTC m=+595.639070421"
Jul 02 13:24:53.179591 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:53.179561    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:24:53.180088 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:53.179718    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:24:53.184138 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:53.184116    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:24:53.184244 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:24:53.184127    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:25:18.427404 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:18.427329    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-9vkdn"
Jul 02 13:25:19.435031 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:19.435005    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:25:20.131454 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.131420    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-jmppc"]
Jul 02 13:25:20.131634 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.131590    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-694f5b7594-jmppc" podUID="3638a2f4-0935-4d00-b448-a9ef5e251c03" containerName="manager" containerID="cri-o://7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80" gracePeriod=10
Jul 02 13:25:20.153720 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.153680    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-9n2bc"]
Jul 02 13:25:20.155896 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.155874    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.167405 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.167385    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-9n2bc"]
Jul 02 13:25:20.286744 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.286689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac594ed-8060-4a56-b0c1-e08b7f70085b-cert\") pod \"kserve-controller-manager-694f5b7594-9n2bc\" (UID: \"5ac594ed-8060-4a56-b0c1-e08b7f70085b\") " pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.286893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.286765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vk5\" (UniqueName: \"kubernetes.io/projected/5ac594ed-8060-4a56-b0c1-e08b7f70085b-kube-api-access-k8vk5\") pod \"kserve-controller-manager-694f5b7594-9n2bc\" (UID: \"5ac594ed-8060-4a56-b0c1-e08b7f70085b\") " pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.366301 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.366280    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:25:20.387777 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.387286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac594ed-8060-4a56-b0c1-e08b7f70085b-cert\") pod \"kserve-controller-manager-694f5b7594-9n2bc\" (UID: \"5ac594ed-8060-4a56-b0c1-e08b7f70085b\") " pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.387777 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.387332    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vk5\" (UniqueName: \"kubernetes.io/projected/5ac594ed-8060-4a56-b0c1-e08b7f70085b-kube-api-access-k8vk5\") pod \"kserve-controller-manager-694f5b7594-9n2bc\" (UID: \"5ac594ed-8060-4a56-b0c1-e08b7f70085b\") " pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.390124 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.390104    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ac594ed-8060-4a56-b0c1-e08b7f70085b-cert\") pod \"kserve-controller-manager-694f5b7594-9n2bc\" (UID: \"5ac594ed-8060-4a56-b0c1-e08b7f70085b\") " pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.396759 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.396734    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vk5\" (UniqueName: \"kubernetes.io/projected/5ac594ed-8060-4a56-b0c1-e08b7f70085b-kube-api-access-k8vk5\") pod \"kserve-controller-manager-694f5b7594-9n2bc\" (UID: \"5ac594ed-8060-4a56-b0c1-e08b7f70085b\") " pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.488644 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.488621    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert\") pod \"3638a2f4-0935-4d00-b448-a9ef5e251c03\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") "
Jul 02 13:25:20.488959 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.488660    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7vbg\" (UniqueName: \"kubernetes.io/projected/3638a2f4-0935-4d00-b448-a9ef5e251c03-kube-api-access-t7vbg\") pod \"3638a2f4-0935-4d00-b448-a9ef5e251c03\" (UID: \"3638a2f4-0935-4d00-b448-a9ef5e251c03\") "
Jul 02 13:25:20.490563 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.490523    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert" (OuterVolumeSpecName: "cert") pod "3638a2f4-0935-4d00-b448-a9ef5e251c03" (UID: "3638a2f4-0935-4d00-b448-a9ef5e251c03"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:25:20.490654 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.490569    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3638a2f4-0935-4d00-b448-a9ef5e251c03-kube-api-access-t7vbg" (OuterVolumeSpecName: "kube-api-access-t7vbg") pod "3638a2f4-0935-4d00-b448-a9ef5e251c03" (UID: "3638a2f4-0935-4d00-b448-a9ef5e251c03"). InnerVolumeSpecName "kube-api-access-t7vbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:25:20.502726 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.502694    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:20.552885 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.552853    2555 generic.go:356] "Generic (PLEG): container finished" podID="3638a2f4-0935-4d00-b448-a9ef5e251c03" containerID="7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80" exitCode=0
Jul 02 13:25:20.553034 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.552899    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-694f5b7594-jmppc"
Jul 02 13:25:20.553130 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.552898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-jmppc" event={"ID":"3638a2f4-0935-4d00-b448-a9ef5e251c03","Type":"ContainerDied","Data":"7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80"}
Jul 02 13:25:20.553356 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.553186    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-jmppc" event={"ID":"3638a2f4-0935-4d00-b448-a9ef5e251c03","Type":"ContainerDied","Data":"3d2e6db637a2769b9f00a4d41280bd28eab0ab21df98b9cd28242c5e63d2fe8f"}
Jul 02 13:25:20.553356 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.553240    2555 scope.go:117] "RemoveContainer" containerID="7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80"
Jul 02 13:25:20.564137 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.564122    2555 scope.go:117] "RemoveContainer" containerID="7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80"
Jul 02 13:25:20.564417 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:25:20.564398    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80\": container with ID starting with 7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80 not found: ID does not exist" containerID="7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80"
Jul 02 13:25:20.564474 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.564425    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80"} err="failed to get container status \"7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80\": rpc error: code = NotFound desc = could not find container \"7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80\": container with ID starting with 7b2439646c6a32db1d0b8377c2cf98f9635c9e12a809f972d9ff4f2879de5f80 not found: ID does not exist"
Jul 02 13:25:20.576414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.576393    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-jmppc"]
Jul 02 13:25:20.580312 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.580292    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-jmppc"]
Jul 02 13:25:20.589451 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.589429    2555 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3638a2f4-0935-4d00-b448-a9ef5e251c03-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:25:20.589531 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.589457    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7vbg\" (UniqueName: \"kubernetes.io/projected/3638a2f4-0935-4d00-b448-a9ef5e251c03-kube-api-access-t7vbg\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:25:20.623163 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:20.623135    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-694f5b7594-9n2bc"]
Jul 02 13:25:20.626160 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:25:20.626141    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac594ed_8060_4a56_b0c1_e08b7f70085b.slice/crio-af530396d0a50fce3191db06f538d79d8a9659eb75d5329bfea1c0637f4adb67 WatchSource:0}: Error finding container af530396d0a50fce3191db06f538d79d8a9659eb75d5329bfea1c0637f4adb67: Status 404 returned error can't find the container with id af530396d0a50fce3191db06f538d79d8a9659eb75d5329bfea1c0637f4adb67
Jul 02 13:25:21.225482 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:21.225445    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3638a2f4-0935-4d00-b448-a9ef5e251c03" path="/var/lib/kubelet/pods/3638a2f4-0935-4d00-b448-a9ef5e251c03/volumes"
Jul 02 13:25:21.558218 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:21.558183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-9n2bc" event={"ID":"5ac594ed-8060-4a56-b0c1-e08b7f70085b","Type":"ContainerStarted","Data":"175b2b17d9768b94be52afa33c2baeac119ef59123aed0e708e1c8fdb3fb6959"}
Jul 02 13:25:21.558685 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:21.558325    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:21.558685 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:21.558368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-694f5b7594-9n2bc" event={"ID":"5ac594ed-8060-4a56-b0c1-e08b7f70085b","Type":"ContainerStarted","Data":"af530396d0a50fce3191db06f538d79d8a9659eb75d5329bfea1c0637f4adb67"}
Jul 02 13:25:21.574299 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:21.574255    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-694f5b7594-9n2bc" podStartSLOduration=1.1724712130000001 podStartE2EDuration="1.574241709s" totalImagesPullingTime="401.770496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:25:21.573643446 +0000 UTC m=+628.763973118" watchObservedRunningTime="2026-07-02 13:25:21.574241709 +0000 UTC m=+628.764571378"
Jul 02 13:25:47.230015 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.229988    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c55ff46-xg8h9"]
Jul 02 13:25:47.230438 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.230361    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3638a2f4-0935-4d00-b448-a9ef5e251c03" containerName="manager"
Jul 02 13:25:47.230438 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.230375    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3638a2f4-0935-4d00-b448-a9ef5e251c03" containerName="manager"
Jul 02 13:25:47.230438 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.230423    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3638a2f4-0935-4d00-b448-a9ef5e251c03" containerName="manager"
Jul 02 13:25:47.232376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.232356    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.245012 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.244992    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c55ff46-xg8h9"]
Jul 02 13:25:47.400165 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.400126    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-service-ca\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.400165 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.400167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-trusted-ca-bundle\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.400376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.400202    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-serving-cert\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.400376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.400218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-oauth-config\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.400376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.400245    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-config\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.400376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.400261    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-oauth-serving-cert\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.400376 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.400282    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4r88\" (UniqueName: \"kubernetes.io/projected/db5d3855-ec4b-4fdb-abf4-dbc131caa490-kube-api-access-d4r88\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.501253 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.501171    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4r88\" (UniqueName: \"kubernetes.io/projected/db5d3855-ec4b-4fdb-abf4-dbc131caa490-kube-api-access-d4r88\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.501253 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.501211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-service-ca\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.501253 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.501234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-trusted-ca-bundle\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.501504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.501263    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-serving-cert\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.501504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.501277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-oauth-config\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.501504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.501311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-config\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.501504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.501340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-oauth-serving-cert\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.507977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.502366    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-oauth-serving-cert\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.507977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.502393    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-service-ca\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.507977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.502490    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-trusted-ca-bundle\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.507977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.502496    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-config\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.507977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.504687    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-oauth-config\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.507977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.504760    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db5d3855-ec4b-4fdb-abf4-dbc131caa490-console-serving-cert\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.509311 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.509289    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4r88\" (UniqueName: \"kubernetes.io/projected/db5d3855-ec4b-4fdb-abf4-dbc131caa490-kube-api-access-d4r88\") pod \"console-6c55ff46-xg8h9\" (UID: \"db5d3855-ec4b-4fdb-abf4-dbc131caa490\") " pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.543284 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.543256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:47.668585 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:47.668561    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c55ff46-xg8h9"]
Jul 02 13:25:47.670946 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:25:47.670919    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5d3855_ec4b_4fdb_abf4_dbc131caa490.slice/crio-3f8aeb13c89d36ecafbd09e3acb7d3b2c60e88054970c425131a74465d9f58b1 WatchSource:0}: Error finding container 3f8aeb13c89d36ecafbd09e3acb7d3b2c60e88054970c425131a74465d9f58b1: Status 404 returned error can't find the container with id 3f8aeb13c89d36ecafbd09e3acb7d3b2c60e88054970c425131a74465d9f58b1
Jul 02 13:25:48.669478 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:48.669444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55ff46-xg8h9" event={"ID":"db5d3855-ec4b-4fdb-abf4-dbc131caa490","Type":"ContainerStarted","Data":"bdf05888fe97eea2227cd7a6498c8bd7f53dc1a4db5f00070bac5295b0689657"}
Jul 02 13:25:48.669876 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:48.669490    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c55ff46-xg8h9" event={"ID":"db5d3855-ec4b-4fdb-abf4-dbc131caa490","Type":"ContainerStarted","Data":"3f8aeb13c89d36ecafbd09e3acb7d3b2c60e88054970c425131a74465d9f58b1"}
Jul 02 13:25:48.687046 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:48.687000    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c55ff46-xg8h9" podStartSLOduration=1.686986452 podStartE2EDuration="1.686986452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:25:48.685678959 +0000 UTC m=+655.876008629" watchObservedRunningTime="2026-07-02 13:25:48.686986452 +0000 UTC m=+655.877316122"
Jul 02 13:25:52.568185 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:52.568153    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-694f5b7594-9n2bc"
Jul 02 13:25:57.543601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:57.543570    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:57.544061 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:57.543854    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:57.548135 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:57.548114    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:57.710636 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:57.710608    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c55ff46-xg8h9"
Jul 02 13:25:57.755685 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:25:57.755656    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58bf85db86-mqlrh"]
Jul 02 13:26:22.782627 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:22.782586    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-58bf85db86-mqlrh" podUID="2549556c-5edb-41a6-848d-7e49da4dc436" containerName="console" containerID="cri-o://8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff" gracePeriod=15
Jul 02 13:26:23.021032 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.021009    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:26:23.064491 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064421    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-oauth-serving-cert\") pod \"2549556c-5edb-41a6-848d-7e49da4dc436\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") "
Jul 02 13:26:23.064491 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064456    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tl9h\" (UniqueName: \"kubernetes.io/projected/2549556c-5edb-41a6-848d-7e49da4dc436-kube-api-access-4tl9h\") pod \"2549556c-5edb-41a6-848d-7e49da4dc436\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") "
Jul 02 13:26:23.064690 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064508    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-oauth-config\") pod \"2549556c-5edb-41a6-848d-7e49da4dc436\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") "
Jul 02 13:26:23.064690 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064532    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-service-ca\") pod \"2549556c-5edb-41a6-848d-7e49da4dc436\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") "
Jul 02 13:26:23.064690 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064577    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-console-config\") pod \"2549556c-5edb-41a6-848d-7e49da4dc436\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") "
Jul 02 13:26:23.064690 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064638    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-serving-cert\") pod \"2549556c-5edb-41a6-848d-7e49da4dc436\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") "
Jul 02 13:26:23.064690 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064669    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-trusted-ca-bundle\") pod \"2549556c-5edb-41a6-848d-7e49da4dc436\" (UID: \"2549556c-5edb-41a6-848d-7e49da4dc436\") "
Jul 02 13:26:23.064959 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064844    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2549556c-5edb-41a6-848d-7e49da4dc436" (UID: "2549556c-5edb-41a6-848d-7e49da4dc436"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:26:23.064959 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.064904    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-oauth-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:26:23.065201 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.065178    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-service-ca" (OuterVolumeSpecName: "service-ca") pod "2549556c-5edb-41a6-848d-7e49da4dc436" (UID: "2549556c-5edb-41a6-848d-7e49da4dc436"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:26:23.065358 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.065213    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2549556c-5edb-41a6-848d-7e49da4dc436" (UID: "2549556c-5edb-41a6-848d-7e49da4dc436"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:26:23.065471 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.065215    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-console-config" (OuterVolumeSpecName: "console-config") pod "2549556c-5edb-41a6-848d-7e49da4dc436" (UID: "2549556c-5edb-41a6-848d-7e49da4dc436"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:26:23.066871 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.066850    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2549556c-5edb-41a6-848d-7e49da4dc436" (UID: "2549556c-5edb-41a6-848d-7e49da4dc436"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:26:23.066999 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.066982    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2549556c-5edb-41a6-848d-7e49da4dc436-kube-api-access-4tl9h" (OuterVolumeSpecName: "kube-api-access-4tl9h") pod "2549556c-5edb-41a6-848d-7e49da4dc436" (UID: "2549556c-5edb-41a6-848d-7e49da4dc436"). InnerVolumeSpecName "kube-api-access-4tl9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:26:23.067053 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.067023    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2549556c-5edb-41a6-848d-7e49da4dc436" (UID: "2549556c-5edb-41a6-848d-7e49da4dc436"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:26:23.166036 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.166013    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-serving-cert\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:26:23.166036 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.166033    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-trusted-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:26:23.166160 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.166042    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tl9h\" (UniqueName: \"kubernetes.io/projected/2549556c-5edb-41a6-848d-7e49da4dc436-kube-api-access-4tl9h\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:26:23.166160 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.166051    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2549556c-5edb-41a6-848d-7e49da4dc436-console-oauth-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:26:23.166160 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.166059    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-service-ca\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:26:23.166160 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.166068    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2549556c-5edb-41a6-848d-7e49da4dc436-console-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:26:23.813264 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.813227    2555 generic.go:356] "Generic (PLEG): container finished" podID="2549556c-5edb-41a6-848d-7e49da4dc436" containerID="8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff" exitCode=0
Jul 02 13:26:23.813264 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.813270    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58bf85db86-mqlrh" event={"ID":"2549556c-5edb-41a6-848d-7e49da4dc436","Type":"ContainerDied","Data":"8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff"}
Jul 02 13:26:23.813800 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.813274    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58bf85db86-mqlrh"
Jul 02 13:26:23.813800 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.813307    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58bf85db86-mqlrh" event={"ID":"2549556c-5edb-41a6-848d-7e49da4dc436","Type":"ContainerDied","Data":"f72c379221a270a6291011f2fb80c9248ced6c6066946946f53f90ad9b07280d"}
Jul 02 13:26:23.813800 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.813333    2555 scope.go:117] "RemoveContainer" containerID="8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff"
Jul 02 13:26:23.821303 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.821273    2555 scope.go:117] "RemoveContainer" containerID="8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff"
Jul 02 13:26:23.821538 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:26:23.821520    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff\": container with ID starting with 8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff not found: ID does not exist" containerID="8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff"
Jul 02 13:26:23.821591 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.821549    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff"} err="failed to get container status \"8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff\": rpc error: code = NotFound desc = could not find container \"8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff\": container with ID starting with 8ba00ffe110473b517a4c1a1d1a88d2cbc29ca51fb20cfbce980a418ed806fff not found: ID does not exist"
Jul 02 13:26:23.829127 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.829106    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58bf85db86-mqlrh"]
Jul 02 13:26:23.831869 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:23.831848    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58bf85db86-mqlrh"]
Jul 02 13:26:25.226092 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:26:25.226061    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2549556c-5edb-41a6-848d-7e49da4dc436" path="/var/lib/kubelet/pods/2549556c-5edb-41a6-848d-7e49da4dc436/volumes"
Jul 02 13:27:40.153382 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.153350    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"]
Jul 02 13:27:40.153879 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.153637    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2549556c-5edb-41a6-848d-7e49da4dc436" containerName="console"
Jul 02 13:27:40.153879 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.153646    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2549556c-5edb-41a6-848d-7e49da4dc436" containerName="console"
Jul 02 13:27:40.153879 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.153715    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2549556c-5edb-41a6-848d-7e49da4dc436" containerName="console"
Jul 02 13:27:40.156789 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.156771    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.159238 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.159215    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 13:27:40.159363 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.159334    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 13:27:40.159363 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.159347    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-acf44-kube-rbac-proxy-sar-config\""
Jul 02 13:27:40.160529 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.160510    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4xk2b\""
Jul 02 13:27:40.160632 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.160517    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-acf44-predictor-serving-cert\""
Jul 02 13:27:40.165624 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.165602    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"]
Jul 02 13:27:40.244128 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.244098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-proxy-tls\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.244128 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.244131    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.244356 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.244156    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsbtt\" (UniqueName: \"kubernetes.io/projected/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-api-access-dsbtt\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.345568 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.345535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-proxy-tls\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.345568 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.345571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.345825 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.345595    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsbtt\" (UniqueName: \"kubernetes.io/projected/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-api-access-dsbtt\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.346200 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.346180    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.348153 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.348131    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-proxy-tls\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.355301 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.355278    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsbtt\" (UniqueName: \"kubernetes.io/projected/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-api-access-dsbtt\") pod \"error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") " pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.467601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.467520    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:40.597165 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:40.597135    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"]
Jul 02 13:27:40.600153 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:27:40.600123    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1b08cc_9488_4eb1_ab3b_6358ea27439a.slice/crio-982641de98c707ee22de21a102e5b0a7b210417cfe5232d44003a35600eb63d0 WatchSource:0}: Error finding container 982641de98c707ee22de21a102e5b0a7b210417cfe5232d44003a35600eb63d0: Status 404 returned error can't find the container with id 982641de98c707ee22de21a102e5b0a7b210417cfe5232d44003a35600eb63d0
Jul 02 13:27:41.123379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:41.123346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" event={"ID":"7b1b08cc-9488-4eb1-ab3b-6358ea27439a","Type":"ContainerStarted","Data":"982641de98c707ee22de21a102e5b0a7b210417cfe5232d44003a35600eb63d0"}
Jul 02 13:27:53.188728 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:53.188679    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" event={"ID":"7b1b08cc-9488-4eb1-ab3b-6358ea27439a","Type":"ContainerStarted","Data":"fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8"}
Jul 02 13:27:55.200503 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:55.200471    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" event={"ID":"7b1b08cc-9488-4eb1-ab3b-6358ea27439a","Type":"ContainerStarted","Data":"1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6"}
Jul 02 13:27:55.200906 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:55.200649    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:55.217357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:55.217313    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podStartSLOduration=0.785318256 podStartE2EDuration="15.217300382s" totalImagesPullingTime="14.431982126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:27:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:27:55.216394673 +0000 UTC m=+782.406724343" watchObservedRunningTime="2026-07-02 13:27:55.217300382 +0000 UTC m=+782.407630052"
Jul 02 13:27:56.204757 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:56.204728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:27:56.205959 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:56.205936    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:27:57.208410 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:27:57.208371    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:28:02.212693 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:28:02.212663    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:28:02.213305 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:28:02.213277    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:28:12.213959 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:28:12.213916    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:28:22.213941 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:28:22.213859    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:28:32.213342 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:28:32.213301    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:28:42.214755 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:28:42.214727    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:29:19.854410 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:19.854378    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"]
Jul 02 13:29:19.856627 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:19.856611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:19.859173 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:19.859150    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-acf44-kube-rbac-proxy-sar-config\""
Jul 02 13:29:19.859368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:19.859156    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-acf44-serving-cert\""
Jul 02 13:29:19.866491 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:19.866469    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"]
Jul 02 13:29:20.039978 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.039946    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls\") pod \"switch-graph-acf44-5486c59c4d-rtpd8\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") " pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.039978 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.039980    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68fbe03-b9cf-49d0-8e32-d2d966afc588-openshift-service-ca-bundle\") pod \"switch-graph-acf44-5486c59c4d-rtpd8\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") " pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.140529 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.140457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls\") pod \"switch-graph-acf44-5486c59c4d-rtpd8\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") " pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.140529 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.140492    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68fbe03-b9cf-49d0-8e32-d2d966afc588-openshift-service-ca-bundle\") pod \"switch-graph-acf44-5486c59c4d-rtpd8\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") " pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.140691 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:29:20.140601    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-acf44-serving-cert: secret "switch-graph-acf44-serving-cert" not found
Jul 02 13:29:20.140691 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:29:20.140663    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls podName:e68fbe03-b9cf-49d0-8e32-d2d966afc588 nodeName:}" failed. No retries permitted until 2026-07-02 13:29:20.640647182 +0000 UTC m=+867.830976833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls") pod "switch-graph-acf44-5486c59c4d-rtpd8" (UID: "e68fbe03-b9cf-49d0-8e32-d2d966afc588") : secret "switch-graph-acf44-serving-cert" not found
Jul 02 13:29:20.141094 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.141078    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68fbe03-b9cf-49d0-8e32-d2d966afc588-openshift-service-ca-bundle\") pod \"switch-graph-acf44-5486c59c4d-rtpd8\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") " pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.645081 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.645047    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls\") pod \"switch-graph-acf44-5486c59c4d-rtpd8\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") " pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.647386 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.647355    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls\") pod \"switch-graph-acf44-5486c59c4d-rtpd8\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") " pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.766121 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.766084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:20.892806 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.892781    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"]
Jul 02 13:29:20.895219 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:29:20.895161    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68fbe03_b9cf_49d0_8e32_d2d966afc588.slice/crio-23f38465e8fcf8eeb0c4edc8cf7f7a9133d7dc7f635bf53fe03bb68b180bdfdb WatchSource:0}: Error finding container 23f38465e8fcf8eeb0c4edc8cf7f7a9133d7dc7f635bf53fe03bb68b180bdfdb: Status 404 returned error can't find the container with id 23f38465e8fcf8eeb0c4edc8cf7f7a9133d7dc7f635bf53fe03bb68b180bdfdb
Jul 02 13:29:20.897220 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:20.897202    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 13:29:21.554884 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:21.554848    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" event={"ID":"e68fbe03-b9cf-49d0-8e32-d2d966afc588","Type":"ContainerStarted","Data":"23f38465e8fcf8eeb0c4edc8cf7f7a9133d7dc7f635bf53fe03bb68b180bdfdb"}
Jul 02 13:29:23.564358 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:23.564272    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" event={"ID":"e68fbe03-b9cf-49d0-8e32-d2d966afc588","Type":"ContainerStarted","Data":"c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb"}
Jul 02 13:29:23.564358 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:23.564356    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:23.579410 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:23.579370    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podStartSLOduration=2.178297159 podStartE2EDuration="4.579354614s" totalImagesPullingTime="2.401057455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:29:23.578321005 +0000 UTC m=+870.768650676" watchObservedRunningTime="2026-07-02 13:29:23.579354614 +0000 UTC m=+870.769684284"
Jul 02 13:29:29.574662 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:29.574624    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:30.105428 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.105382    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"]
Jul 02 13:29:30.105653 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.105598    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44" containerID="cri-o://c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb" gracePeriod=30
Jul 02 13:29:30.335840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.335813    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"]
Jul 02 13:29:30.338039 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.338025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.340432 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.340412    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-383ed-predictor-serving-cert\""
Jul 02 13:29:30.340432 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.340423    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-383ed-kube-rbac-proxy-sar-config\""
Jul 02 13:29:30.346743 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.346696    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"]
Jul 02 13:29:30.390629 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.390556    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"]
Jul 02 13:29:30.390823 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.390796    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" containerID="cri-o://fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8" gracePeriod=30
Jul 02 13:29:30.390907 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.390844    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kube-rbac-proxy" containerID="cri-o://1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6" gracePeriod=30
Jul 02 13:29:30.414292 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.414265    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba78be6a-6f3b-419c-97dc-1c86c90721a1-proxy-tls\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.414398 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.414312    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.414398 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.414344    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhl2\" (UniqueName: \"kubernetes.io/projected/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-api-access-gvhl2\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.515641 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.515614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.515761 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.515657    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhl2\" (UniqueName: \"kubernetes.io/projected/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-api-access-gvhl2\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.515761 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.515714    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba78be6a-6f3b-419c-97dc-1c86c90721a1-proxy-tls\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.516209 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.516190    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.517738 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.517696    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba78be6a-6f3b-419c-97dc-1c86c90721a1-proxy-tls\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.522900 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.522876    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhl2\" (UniqueName: \"kubernetes.io/projected/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-api-access-gvhl2\") pod \"error-404-isvc-383ed-predictor-6fbb59466b-2252r\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") " pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.593506 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.593479    2555 generic.go:356] "Generic (PLEG): container finished" podID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerID="1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6" exitCode=2
Jul 02 13:29:30.593843 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.593553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" event={"ID":"7b1b08cc-9488-4eb1-ab3b-6358ea27439a","Type":"ContainerDied","Data":"1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6"}
Jul 02 13:29:30.649023 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.648969    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:30.770780 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:30.770752    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"]
Jul 02 13:29:30.772986 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:29:30.772959    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba78be6a_6f3b_419c_97dc_1c86c90721a1.slice/crio-f2951af519119b7063a06ad536c85bd25058d9d6d769f3dea51b57c3f5c35b02 WatchSource:0}: Error finding container f2951af519119b7063a06ad536c85bd25058d9d6d769f3dea51b57c3f5c35b02: Status 404 returned error can't find the container with id f2951af519119b7063a06ad536c85bd25058d9d6d769f3dea51b57c3f5c35b02
Jul 02 13:29:31.599433 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:31.599400    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" event={"ID":"ba78be6a-6f3b-419c-97dc-1c86c90721a1","Type":"ContainerStarted","Data":"90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d"}
Jul 02 13:29:31.599834 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:31.599456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" event={"ID":"ba78be6a-6f3b-419c-97dc-1c86c90721a1","Type":"ContainerStarted","Data":"3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9"}
Jul 02 13:29:31.599834 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:31.599480    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" event={"ID":"ba78be6a-6f3b-419c-97dc-1c86c90721a1","Type":"ContainerStarted","Data":"f2951af519119b7063a06ad536c85bd25058d9d6d769f3dea51b57c3f5c35b02"}
Jul 02 13:29:31.599834 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:31.599537    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:31.616251 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:31.616208    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podStartSLOduration=1.616194924 podStartE2EDuration="1.616194924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:29:31.61461923 +0000 UTC m=+878.804948900" watchObservedRunningTime="2026-07-02 13:29:31.616194924 +0000 UTC m=+878.806524593"
Jul 02 13:29:32.209326 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:32.209290    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 02 13:29:32.213574 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:32.213551    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 02 13:29:32.604159 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:32.604129    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:32.605329 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:32.605303    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:29:33.246942 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.246921    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:29:33.339856 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.339777    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-proxy-tls\") pod \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") "
Jul 02 13:29:33.339856 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.339852    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-rbac-proxy-sar-config\") pod \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") "
Jul 02 13:29:33.340054 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.339939    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsbtt\" (UniqueName: \"kubernetes.io/projected/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-api-access-dsbtt\") pod \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\" (UID: \"7b1b08cc-9488-4eb1-ab3b-6358ea27439a\") "
Jul 02 13:29:33.340251 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.340221    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7b1b08cc-9488-4eb1-ab3b-6358ea27439a" (UID: "7b1b08cc-9488-4eb1-ab3b-6358ea27439a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:29:33.341940 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.341913    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b1b08cc-9488-4eb1-ab3b-6358ea27439a" (UID: "7b1b08cc-9488-4eb1-ab3b-6358ea27439a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:29:33.342066 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.342012    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-api-access-dsbtt" (OuterVolumeSpecName: "kube-api-access-dsbtt") pod "7b1b08cc-9488-4eb1-ab3b-6358ea27439a" (UID: "7b1b08cc-9488-4eb1-ab3b-6358ea27439a"). InnerVolumeSpecName "kube-api-access-dsbtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:29:33.441115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.441086    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:29:33.441115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.441109    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:29:33.441115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.441120    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsbtt\" (UniqueName: \"kubernetes.io/projected/7b1b08cc-9488-4eb1-ab3b-6358ea27439a-kube-api-access-dsbtt\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:29:33.610103 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.610020    2555 generic.go:356] "Generic (PLEG): container finished" podID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerID="fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8" exitCode=0
Jul 02 13:29:33.610103 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.610084    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"
Jul 02 13:29:33.610554 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.610098    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" event={"ID":"7b1b08cc-9488-4eb1-ab3b-6358ea27439a","Type":"ContainerDied","Data":"fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8"}
Jul 02 13:29:33.610554 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.610180    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h" event={"ID":"7b1b08cc-9488-4eb1-ab3b-6358ea27439a","Type":"ContainerDied","Data":"982641de98c707ee22de21a102e5b0a7b210417cfe5232d44003a35600eb63d0"}
Jul 02 13:29:33.610554 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.610207    2555 scope.go:117] "RemoveContainer" containerID="1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6"
Jul 02 13:29:33.610554 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.610470    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:29:33.621660 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.621640    2555 scope.go:117] "RemoveContainer" containerID="fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8"
Jul 02 13:29:33.629647 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.629625    2555 scope.go:117] "RemoveContainer" containerID="1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6"
Jul 02 13:29:33.629922 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:29:33.629902    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6\": container with ID starting with 1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6 not found: ID does not exist" containerID="1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6"
Jul 02 13:29:33.630005 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.629932    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6"} err="failed to get container status \"1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6\": rpc error: code = NotFound desc = could not find container \"1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6\": container with ID starting with 1e4159516330ee451763b1592b01e8cfd1d5caacf6cdbcb8325a8f9c9eb30ee6 not found: ID does not exist"
Jul 02 13:29:33.630005 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.629953    2555 scope.go:117] "RemoveContainer" containerID="fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8"
Jul 02 13:29:33.630190 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:29:33.630173    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8\": container with ID starting with fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8 not found: ID does not exist" containerID="fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8"
Jul 02 13:29:33.630229 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.630195    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8"} err="failed to get container status \"fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8\": rpc error: code = NotFound desc = could not find container \"fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8\": container with ID starting with fa55ca9dda96260591b35570a6cf5360a528c0bc07c334cef230e7307dd50bd8 not found: ID does not exist"
Jul 02 13:29:33.634246 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.634222    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"]
Jul 02 13:29:33.639698 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:33.639680    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acf44-predictor-7bdc57d7d4-bsv9h"]
Jul 02 13:29:34.571917 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:34.571882    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:35.225058 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:35.225028    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" path="/var/lib/kubelet/pods/7b1b08cc-9488-4eb1-ab3b-6358ea27439a/volumes"
Jul 02 13:29:38.614567 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:38.614544    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:29:38.615007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:38.614981    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:29:39.572152 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:39.572115    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:44.572419 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:44.572334    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:44.572828 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:44.572431    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:29:48.615074 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:48.615037    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:29:49.572666 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:49.572627    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:53.209588 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:53.209559    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:29:53.210021 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:53.209753    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:29:53.213774 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:53.213754    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:29:53.213894 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:53.213878    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:29:54.572474 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:54.572429    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:58.615924 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:58.615888    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:29:59.572289 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.572247    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:29:59.823629 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.823555    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"]
Jul 02 13:29:59.823990 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.823879    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container"
Jul 02 13:29:59.823990 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.823890    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container"
Jul 02 13:29:59.823990 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.823897    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kube-rbac-proxy"
Jul 02 13:29:59.823990 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.823903    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kube-rbac-proxy"
Jul 02 13:29:59.823990 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.823958    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kserve-container"
Jul 02 13:29:59.823990 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.823969    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b1b08cc-9488-4eb1-ab3b-6358ea27439a" containerName="kube-rbac-proxy"
Jul 02 13:29:59.826882 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.826866    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:29:59.829399 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.829375    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 02 13:29:59.829399 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.829375    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 02 13:29:59.836285 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.836266    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"]
Jul 02 13:29:59.951986 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.951953    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-openshift-service-ca-bundle\") pod \"model-chainer-9d455c95-bnm99\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") " pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:29:59.951986 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:29:59.951992    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-proxy-tls\") pod \"model-chainer-9d455c95-bnm99\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") " pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:00.053357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.053318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-openshift-service-ca-bundle\") pod \"model-chainer-9d455c95-bnm99\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") " pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:00.053357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.053358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-proxy-tls\") pod \"model-chainer-9d455c95-bnm99\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") " pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:00.053961 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.053941    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-openshift-service-ca-bundle\") pod \"model-chainer-9d455c95-bnm99\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") " pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:00.055609 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.055587    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-proxy-tls\") pod \"model-chainer-9d455c95-bnm99\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") " pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:00.136996 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.136954    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:00.257642 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.257617    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:30:00.268161 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.268142    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"]
Jul 02 13:30:00.270098 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:30:00.270081    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7d32fc_a9c5_4896_a945_96ae0a7f4d5f.slice/crio-ab4843b52d1e4bdd4322d37f8b7ecb098732f67f3b43beecbf1ce10fae0f686d WatchSource:0}: Error finding container ab4843b52d1e4bdd4322d37f8b7ecb098732f67f3b43beecbf1ce10fae0f686d: Status 404 returned error can't find the container with id ab4843b52d1e4bdd4322d37f8b7ecb098732f67f3b43beecbf1ce10fae0f686d
Jul 02 13:30:00.355253 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.355226    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68fbe03-b9cf-49d0-8e32-d2d966afc588-openshift-service-ca-bundle\") pod \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") "
Jul 02 13:30:00.355397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.355318    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls\") pod \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\" (UID: \"e68fbe03-b9cf-49d0-8e32-d2d966afc588\") "
Jul 02 13:30:00.355581 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.355559    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68fbe03-b9cf-49d0-8e32-d2d966afc588-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e68fbe03-b9cf-49d0-8e32-d2d966afc588" (UID: "e68fbe03-b9cf-49d0-8e32-d2d966afc588"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:30:00.357142 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.357124    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e68fbe03-b9cf-49d0-8e32-d2d966afc588" (UID: "e68fbe03-b9cf-49d0-8e32-d2d966afc588"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:30:00.456831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.456752    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e68fbe03-b9cf-49d0-8e32-d2d966afc588-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:30:00.456831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.456779    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e68fbe03-b9cf-49d0-8e32-d2d966afc588-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:30:00.727654 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.727566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" event={"ID":"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f","Type":"ContainerStarted","Data":"dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b"}
Jul 02 13:30:00.727654 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.727624    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" event={"ID":"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f","Type":"ContainerStarted","Data":"ab4843b52d1e4bdd4322d37f8b7ecb098732f67f3b43beecbf1ce10fae0f686d"}
Jul 02 13:30:00.727893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.727655    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:00.728746 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.728723    2555 generic.go:356] "Generic (PLEG): container finished" podID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerID="c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb" exitCode=0
Jul 02 13:30:00.728840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.728753    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"
Jul 02 13:30:00.728840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.728771    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" event={"ID":"e68fbe03-b9cf-49d0-8e32-d2d966afc588","Type":"ContainerDied","Data":"c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb"}
Jul 02 13:30:00.728840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.728806    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8" event={"ID":"e68fbe03-b9cf-49d0-8e32-d2d966afc588","Type":"ContainerDied","Data":"23f38465e8fcf8eeb0c4edc8cf7f7a9133d7dc7f635bf53fe03bb68b180bdfdb"}
Jul 02 13:30:00.728840 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.728827    2555 scope.go:117] "RemoveContainer" containerID="c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb"
Jul 02 13:30:00.737754 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.737663    2555 scope.go:117] "RemoveContainer" containerID="c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb"
Jul 02 13:30:00.739424 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:30:00.739403    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb\": container with ID starting with c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb not found: ID does not exist" containerID="c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb"
Jul 02 13:30:00.739508 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.739432    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb"} err="failed to get container status \"c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb\": rpc error: code = NotFound desc = could not find container \"c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb\": container with ID starting with c313e6441d6b73add4f9ed5c96f7def733aaff102005b3c5f1dcfa8ac28969bb not found: ID does not exist"
Jul 02 13:30:00.745883 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.745847    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podStartSLOduration=1.74583397 podStartE2EDuration="1.74583397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:30:00.74518397 +0000 UTC m=+907.935513640" watchObservedRunningTime="2026-07-02 13:30:00.74583397 +0000 UTC m=+907.936163638"
Jul 02 13:30:00.759592 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.759569    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"]
Jul 02 13:30:00.765067 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:00.765048    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-acf44-5486c59c4d-rtpd8"]
Jul 02 13:30:01.225146 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:01.225116    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" path="/var/lib/kubelet/pods/e68fbe03-b9cf-49d0-8e32-d2d966afc588/volumes"
Jul 02 13:30:06.740650 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:06.740623    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:08.615531 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:08.615491    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 02 13:30:09.925857 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:09.925825    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"]
Jul 02 13:30:09.926328 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:09.926069    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer" containerID="cri-o://dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b" gracePeriod=30
Jul 02 13:30:10.166733 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.166673    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"]
Jul 02 13:30:10.167084 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.167068    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44"
Jul 02 13:30:10.167188 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.167087    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44"
Jul 02 13:30:10.167246 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.167186    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68fbe03-b9cf-49d0-8e32-d2d966afc588" containerName="switch-graph-acf44"
Jul 02 13:30:10.170509 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.170489    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.172970 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.172949    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f369f-predictor-serving-cert\""
Jul 02 13:30:10.173045 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.172949    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f369f-kube-rbac-proxy-sar-config\""
Jul 02 13:30:10.178780 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.178761    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"]
Jul 02 13:30:10.338758 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.338722    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nvs\" (UniqueName: \"kubernetes.io/projected/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-api-access-h8nvs\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.338922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.338795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-proxy-tls\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.338922 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.338888    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.439548 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.439468    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nvs\" (UniqueName: \"kubernetes.io/projected/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-api-access-h8nvs\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.439548 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.439520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-proxy-tls\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.439768 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.439559    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.440277 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.440258    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.442039 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.442015    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-proxy-tls\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.447759 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.447738    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nvs\" (UniqueName: \"kubernetes.io/projected/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-api-access-h8nvs\") pod \"error-404-isvc-f369f-predictor-784754f556-n6j5x\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") " pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.481582 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.481557    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.605292 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.605266    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"]
Jul 02 13:30:10.607300 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:30:10.607270    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca3c6a8_3ce7_469a_bf23_76cea2611e36.slice/crio-c69a1bd3134f79fe020ba904f3181c1a763a07de4f252e17a885b1faf1a5ed1e WatchSource:0}: Error finding container c69a1bd3134f79fe020ba904f3181c1a763a07de4f252e17a885b1faf1a5ed1e: Status 404 returned error can't find the container with id c69a1bd3134f79fe020ba904f3181c1a763a07de4f252e17a885b1faf1a5ed1e
Jul 02 13:30:10.774664 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.774628    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" event={"ID":"3ca3c6a8-3ce7-469a-bf23-76cea2611e36","Type":"ContainerStarted","Data":"50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c"}
Jul 02 13:30:10.774917 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.774693    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" event={"ID":"3ca3c6a8-3ce7-469a-bf23-76cea2611e36","Type":"ContainerStarted","Data":"bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962"}
Jul 02 13:30:10.774917 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.774745    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" event={"ID":"3ca3c6a8-3ce7-469a-bf23-76cea2611e36","Type":"ContainerStarted","Data":"c69a1bd3134f79fe020ba904f3181c1a763a07de4f252e17a885b1faf1a5ed1e"}
Jul 02 13:30:10.774917 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.774893    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.775078 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.774944    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:10.776330 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.776302    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:30:10.790820 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:10.790765    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podStartSLOduration=0.790752707 podStartE2EDuration="790.752707ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:30:10.789606487 +0000 UTC m=+917.979936157" watchObservedRunningTime="2026-07-02 13:30:10.790752707 +0000 UTC m=+917.981082371"
Jul 02 13:30:11.739190 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:11.739151    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:30:11.779367 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:11.779331    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:30:16.739195 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:16.739156    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:30:16.785044 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:16.785015    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:30:16.785423 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:16.785395    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:30:18.615651 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:18.615624    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:30:21.738648 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:21.738603    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:30:21.739038 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:21.738699    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:26.739362 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:26.739318    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:30:26.786084 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:26.786051    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:30:30.316415 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.316386    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"]
Jul 02 13:30:30.319928 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.319912    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:30.322311 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.322289    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-383ed-serving-cert\""
Jul 02 13:30:30.322404 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.322388    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-383ed-kube-rbac-proxy-sar-config\""
Jul 02 13:30:30.328051 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.328028    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"]
Jul 02 13:30:30.392378 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.392321    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls\") pod \"switch-graph-383ed-79cc5cfd9b-cjs9c\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") " pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:30.392566 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.392418    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d54e0e73-1c73-430a-a65a-aa0c530d90d4-openshift-service-ca-bundle\") pod \"switch-graph-383ed-79cc5cfd9b-cjs9c\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") " pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:30.492968 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.492935    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls\") pod \"switch-graph-383ed-79cc5cfd9b-cjs9c\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") " pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:30.492968 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.492982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d54e0e73-1c73-430a-a65a-aa0c530d90d4-openshift-service-ca-bundle\") pod \"switch-graph-383ed-79cc5cfd9b-cjs9c\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") " pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:30.493168 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:30:30.493100    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-383ed-serving-cert: secret "switch-graph-383ed-serving-cert" not found
Jul 02 13:30:30.493168 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:30:30.493167    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls podName:d54e0e73-1c73-430a-a65a-aa0c530d90d4 nodeName:}" failed. No retries permitted until 2026-07-02 13:30:30.993147962 +0000 UTC m=+938.183477625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls") pod "switch-graph-383ed-79cc5cfd9b-cjs9c" (UID: "d54e0e73-1c73-430a-a65a-aa0c530d90d4") : secret "switch-graph-383ed-serving-cert" not found
Jul 02 13:30:30.493659 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.493640    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d54e0e73-1c73-430a-a65a-aa0c530d90d4-openshift-service-ca-bundle\") pod \"switch-graph-383ed-79cc5cfd9b-cjs9c\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") " pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:30.997556 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.997518    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls\") pod \"switch-graph-383ed-79cc5cfd9b-cjs9c\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") " pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:30.999781 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:30.999762    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls\") pod \"switch-graph-383ed-79cc5cfd9b-cjs9c\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") " pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:31.231863 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:31.231836    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:31.357951 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:31.357772    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"]
Jul 02 13:30:31.360883 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:30:31.360840    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54e0e73_1c73_430a_a65a_aa0c530d90d4.slice/crio-823acdf8cc0970ced5e4f0e9762b3a07e3f3f9ca90fa84652f0429dd2a4e3229 WatchSource:0}: Error finding container 823acdf8cc0970ced5e4f0e9762b3a07e3f3f9ca90fa84652f0429dd2a4e3229: Status 404 returned error can't find the container with id 823acdf8cc0970ced5e4f0e9762b3a07e3f3f9ca90fa84652f0429dd2a4e3229
Jul 02 13:30:31.738768 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:31.738733    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:30:31.864724 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:31.864666    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" event={"ID":"d54e0e73-1c73-430a-a65a-aa0c530d90d4","Type":"ContainerStarted","Data":"d2a736f4501430f1bed1ee0ccbe70dfcf33c1eeb62f67b2b75788a7c9842276a"}
Jul 02 13:30:31.864913 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:31.864743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" event={"ID":"d54e0e73-1c73-430a-a65a-aa0c530d90d4","Type":"ContainerStarted","Data":"823acdf8cc0970ced5e4f0e9762b3a07e3f3f9ca90fa84652f0429dd2a4e3229"}
Jul 02 13:30:31.864913 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:31.864779    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:31.880758 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:31.880697    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podStartSLOduration=1.880682588 podStartE2EDuration="1.880682588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:30:31.879326467 +0000 UTC m=+939.069656138" watchObservedRunningTime="2026-07-02 13:30:31.880682588 +0000 UTC m=+939.071012258"
Jul 02 13:30:36.739271 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:36.739225    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:30:36.785916 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:36.785884    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:30:37.875563 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:37.875537    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:30:40.571435 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.571410    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:40.670776 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.670746    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-openshift-service-ca-bundle\") pod \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") "
Jul 02 13:30:40.670776 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.670777    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-proxy-tls\") pod \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\" (UID: \"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f\") "
Jul 02 13:30:40.671100 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.671079    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" (UID: "2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:30:40.672744 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.672694    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" (UID: "2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:30:40.771945 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.771865    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:30:40.771945 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.771893    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:30:40.905256 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.905218    2555 generic.go:356] "Generic (PLEG): container finished" podID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerID="dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b" exitCode=0
Jul 02 13:30:40.905420 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.905262    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"
Jul 02 13:30:40.905420 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.905269    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" event={"ID":"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f","Type":"ContainerDied","Data":"dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b"}
Jul 02 13:30:40.905420 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.905322    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-9d455c95-bnm99" event={"ID":"2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f","Type":"ContainerDied","Data":"ab4843b52d1e4bdd4322d37f8b7ecb098732f67f3b43beecbf1ce10fae0f686d"}
Jul 02 13:30:40.905420 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.905351    2555 scope.go:117] "RemoveContainer" containerID="dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b"
Jul 02 13:30:40.914459 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.914442    2555 scope.go:117] "RemoveContainer" containerID="dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b"
Jul 02 13:30:40.914698 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:30:40.914682    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b\": container with ID starting with dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b not found: ID does not exist" containerID="dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b"
Jul 02 13:30:40.914772 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.914727    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b"} err="failed to get container status \"dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b\": rpc error: code = NotFound desc = could not find container \"dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b\": container with ID starting with dd4fe85194b87a0501089c49bdeae45ba43f2d889f4f3807d846066583a9e77b not found: ID does not exist"
Jul 02 13:30:40.925887 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.925864    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"]
Jul 02 13:30:40.929404 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:40.929385    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-9d455c95-bnm99"]
Jul 02 13:30:41.225216 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:41.225181    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" path="/var/lib/kubelet/pods/2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f/volumes"
Jul 02 13:30:46.785580 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:46.785543    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:30:56.785887 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:30:56.785856    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:31:10.092092 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.092061    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"]
Jul 02 13:31:10.092512 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.092393    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer"
Jul 02 13:31:10.092512 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.092404    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer"
Jul 02 13:31:10.092512 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.092467    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7d32fc-a9c5-4896-a945-96ae0a7f4d5f" containerName="model-chainer"
Jul 02 13:31:10.096731 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.096696    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.099320 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.099295    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f369f-serving-cert\""
Jul 02 13:31:10.099439 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.099364    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f369f-kube-rbac-proxy-sar-config\""
Jul 02 13:31:10.104099 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.104072    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"]
Jul 02 13:31:10.208442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.208412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fb52c1-e059-4a93-9aea-3468ebc711f6-openshift-service-ca-bundle\") pod \"sequence-graph-f369f-d9b69b6db-2jn9z\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") " pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.208601 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.208450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fb52c1-e059-4a93-9aea-3468ebc711f6-proxy-tls\") pod \"sequence-graph-f369f-d9b69b6db-2jn9z\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") " pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.309123 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.309099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fb52c1-e059-4a93-9aea-3468ebc711f6-openshift-service-ca-bundle\") pod \"sequence-graph-f369f-d9b69b6db-2jn9z\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") " pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.309269 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.309130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fb52c1-e059-4a93-9aea-3468ebc711f6-proxy-tls\") pod \"sequence-graph-f369f-d9b69b6db-2jn9z\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") " pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.309676 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.309647    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fb52c1-e059-4a93-9aea-3468ebc711f6-openshift-service-ca-bundle\") pod \"sequence-graph-f369f-d9b69b6db-2jn9z\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") " pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.311371 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.311355    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fb52c1-e059-4a93-9aea-3468ebc711f6-proxy-tls\") pod \"sequence-graph-f369f-d9b69b6db-2jn9z\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") " pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.407548 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.407488    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:10.527049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:10.526958    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"]
Jul 02 13:31:10.529140 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:31:10.529110    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fb52c1_e059_4a93_9aea_3468ebc711f6.slice/crio-617b8b6d04ceb6604fbb21c1caf5df89043120dec72706e7b144a5643b0530b4 WatchSource:0}: Error finding container 617b8b6d04ceb6604fbb21c1caf5df89043120dec72706e7b144a5643b0530b4: Status 404 returned error can't find the container with id 617b8b6d04ceb6604fbb21c1caf5df89043120dec72706e7b144a5643b0530b4
Jul 02 13:31:11.041948 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:11.041916    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" event={"ID":"07fb52c1-e059-4a93-9aea-3468ebc711f6","Type":"ContainerStarted","Data":"44b95d92f82e7217244074768ec6c82cd8418c62051cfe3576c21f354bd3e554"}
Jul 02 13:31:11.042122 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:11.041973    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" event={"ID":"07fb52c1-e059-4a93-9aea-3468ebc711f6","Type":"ContainerStarted","Data":"617b8b6d04ceb6604fbb21c1caf5df89043120dec72706e7b144a5643b0530b4"}
Jul 02 13:31:11.042122 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:11.042007    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:31:11.056863 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:11.056818    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podStartSLOduration=1.056804752 podStartE2EDuration="1.056804752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:31:11.055918017 +0000 UTC m=+978.246247686" watchObservedRunningTime="2026-07-02 13:31:11.056804752 +0000 UTC m=+978.247134444"
Jul 02 13:31:17.054003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:31:17.053929    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:34:53.240492 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:34:53.240468    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:34:53.241012 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:34:53.240901    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:34:53.244746 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:34:53.244725    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:34:53.244912 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:34:53.244899    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:38:45.008613 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.008530    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"]
Jul 02 13:38:45.009188 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.008812    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed" containerID="cri-o://d2a736f4501430f1bed1ee0ccbe70dfcf33c1eeb62f67b2b75788a7c9842276a" gracePeriod=30
Jul 02 13:38:45.241375 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.241344    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"]
Jul 02 13:38:45.245028 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.245008    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.247480 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.247457    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6c5e8-predictor-serving-cert\""
Jul 02 13:38:45.247480 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.247469    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6c5e8-kube-rbac-proxy-sar-config\""
Jul 02 13:38:45.254464 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.254445    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"]
Jul 02 13:38:45.286669 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.286596    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"]
Jul 02 13:38:45.286874 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.286850    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container" containerID="cri-o://3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9" gracePeriod=30
Jul 02 13:38:45.287075 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.286940    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kube-rbac-proxy" containerID="cri-o://90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d" gracePeriod=30
Jul 02 13:38:45.364357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.364334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4gq\" (UniqueName: \"kubernetes.io/projected/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-api-access-jm4gq\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.364507 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.364484    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.364584 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.364531    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcd366e1-f14f-4f7b-846c-987e5b4b5298-proxy-tls\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.465054 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.465018    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.465222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.465067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcd366e1-f14f-4f7b-846c-987e5b4b5298-proxy-tls\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.465222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.465101    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4gq\" (UniqueName: \"kubernetes.io/projected/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-api-access-jm4gq\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.465652 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.465632    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.467359 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.467340    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcd366e1-f14f-4f7b-846c-987e5b4b5298-proxy-tls\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.472491 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.472470    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4gq\" (UniqueName: \"kubernetes.io/projected/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-api-access-jm4gq\") pod \"error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") " pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.555652 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.555589    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.681812 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.681790    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"]
Jul 02 13:38:45.683886 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:38:45.683857    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd366e1_f14f_4f7b_846c_987e5b4b5298.slice/crio-8247aeaf7dc1fdfb4e6524b5520009e6ace47593652daccced6c7318802e3649 WatchSource:0}: Error finding container 8247aeaf7dc1fdfb4e6524b5520009e6ace47593652daccced6c7318802e3649: Status 404 returned error can't find the container with id 8247aeaf7dc1fdfb4e6524b5520009e6ace47593652daccced6c7318802e3649
Jul 02 13:38:45.957730 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.957677    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" event={"ID":"fcd366e1-f14f-4f7b-846c-987e5b4b5298","Type":"ContainerStarted","Data":"f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3"}
Jul 02 13:38:45.957884 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.957765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" event={"ID":"fcd366e1-f14f-4f7b-846c-987e5b4b5298","Type":"ContainerStarted","Data":"0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a"}
Jul 02 13:38:45.957884 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.957805    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:45.957884 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.957832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" event={"ID":"fcd366e1-f14f-4f7b-846c-987e5b4b5298","Type":"ContainerStarted","Data":"8247aeaf7dc1fdfb4e6524b5520009e6ace47593652daccced6c7318802e3649"}
Jul 02 13:38:45.959489 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.959464    2555 generic.go:356] "Generic (PLEG): container finished" podID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerID="90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d" exitCode=2
Jul 02 13:38:45.959608 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.959520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" event={"ID":"ba78be6a-6f3b-419c-97dc-1c86c90721a1","Type":"ContainerDied","Data":"90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d"}
Jul 02 13:38:45.974743 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:45.974680    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podStartSLOduration=0.97466752 podStartE2EDuration="974.66752ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:38:45.974044051 +0000 UTC m=+1433.164373722" watchObservedRunningTime="2026-07-02 13:38:45.97466752 +0000 UTC m=+1433.164997190"
Jul 02 13:38:46.964306 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:46.964274    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:46.965683 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:46.965655    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:38:47.874081 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:47.874042    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:38:47.968691 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:47.968650    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:38:48.235538 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.235511    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:38:48.389011 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.388980    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba78be6a-6f3b-419c-97dc-1c86c90721a1-proxy-tls\") pod \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") "
Jul 02 13:38:48.389169 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.389024    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-rbac-proxy-sar-config\") pod \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") "
Jul 02 13:38:48.389169 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.389057    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvhl2\" (UniqueName: \"kubernetes.io/projected/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-api-access-gvhl2\") pod \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\" (UID: \"ba78be6a-6f3b-419c-97dc-1c86c90721a1\") "
Jul 02 13:38:48.389439 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.389408    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba78be6a-6f3b-419c-97dc-1c86c90721a1" (UID: "ba78be6a-6f3b-419c-97dc-1c86c90721a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:38:48.391064 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.391032    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba78be6a-6f3b-419c-97dc-1c86c90721a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba78be6a-6f3b-419c-97dc-1c86c90721a1" (UID: "ba78be6a-6f3b-419c-97dc-1c86c90721a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:38:48.391159 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.391089    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-api-access-gvhl2" (OuterVolumeSpecName: "kube-api-access-gvhl2") pod "ba78be6a-6f3b-419c-97dc-1c86c90721a1" (UID: "ba78be6a-6f3b-419c-97dc-1c86c90721a1"). InnerVolumeSpecName "kube-api-access-gvhl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:38:48.490117 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.490038    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba78be6a-6f3b-419c-97dc-1c86c90721a1-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:38:48.490117 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.490076    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:38:48.490117 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.490091    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhl2\" (UniqueName: \"kubernetes.io/projected/ba78be6a-6f3b-419c-97dc-1c86c90721a1-kube-api-access-gvhl2\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:38:48.975442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.975407    2555 generic.go:356] "Generic (PLEG): container finished" podID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerID="3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9" exitCode=0
Jul 02 13:38:48.975805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.975472    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"
Jul 02 13:38:48.975805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.975495    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" event={"ID":"ba78be6a-6f3b-419c-97dc-1c86c90721a1","Type":"ContainerDied","Data":"3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9"}
Jul 02 13:38:48.975805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.975580    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r" event={"ID":"ba78be6a-6f3b-419c-97dc-1c86c90721a1","Type":"ContainerDied","Data":"f2951af519119b7063a06ad536c85bd25058d9d6d769f3dea51b57c3f5c35b02"}
Jul 02 13:38:48.975805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.975606    2555 scope.go:117] "RemoveContainer" containerID="90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d"
Jul 02 13:38:48.984657 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.984642    2555 scope.go:117] "RemoveContainer" containerID="3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9"
Jul 02 13:38:48.992024 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.992000    2555 scope.go:117] "RemoveContainer" containerID="90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d"
Jul 02 13:38:48.992262 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:38:48.992242    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d\": container with ID starting with 90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d not found: ID does not exist" containerID="90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d"
Jul 02 13:38:48.992314 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.992269    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d"} err="failed to get container status \"90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d\": rpc error: code = NotFound desc = could not find container \"90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d\": container with ID starting with 90b0f6c00582813c1ae854918907e07fb018428d614a8385f0b999934c01287d not found: ID does not exist"
Jul 02 13:38:48.992314 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.992287    2555 scope.go:117] "RemoveContainer" containerID="3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9"
Jul 02 13:38:48.992517 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:38:48.992499    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9\": container with ID starting with 3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9 not found: ID does not exist" containerID="3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9"
Jul 02 13:38:48.992575 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.992524    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9"} err="failed to get container status \"3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9\": rpc error: code = NotFound desc = could not find container \"3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9\": container with ID starting with 3a0a299fd9451b97916a2d6bc679a6ef4021233ffd1f107b149f66d7ec63f2f9 not found: ID does not exist"
Jul 02 13:38:48.996727 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.996688    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"]
Jul 02 13:38:48.999883 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:48.999865    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-383ed-predictor-6fbb59466b-2252r"]
Jul 02 13:38:49.226561 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:49.226471    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" path="/var/lib/kubelet/pods/ba78be6a-6f3b-419c-97dc-1c86c90721a1/volumes"
Jul 02 13:38:52.873486 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:52.873449    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:38:52.972754 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:52.972734    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:38:52.973145 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:52.973122    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:38:57.874359 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:57.874314    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:38:57.874786 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:38:57.874432    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:39:02.874166 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:02.874116    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:02.973681 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:02.973644    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:39:07.874553 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:07.874505    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:12.874018 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:12.873976    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:12.973526 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:12.973493    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:39:15.093672 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.093643    2555 generic.go:356] "Generic (PLEG): container finished" podID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerID="d2a736f4501430f1bed1ee0ccbe70dfcf33c1eeb62f67b2b75788a7c9842276a" exitCode=0
Jul 02 13:39:15.093991 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.093740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" event={"ID":"d54e0e73-1c73-430a-a65a-aa0c530d90d4","Type":"ContainerDied","Data":"d2a736f4501430f1bed1ee0ccbe70dfcf33c1eeb62f67b2b75788a7c9842276a"}
Jul 02 13:39:15.158062 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.158043    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:39:15.206948 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.206919    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls\") pod \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") "
Jul 02 13:39:15.207064 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.207004    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d54e0e73-1c73-430a-a65a-aa0c530d90d4-openshift-service-ca-bundle\") pod \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\" (UID: \"d54e0e73-1c73-430a-a65a-aa0c530d90d4\") "
Jul 02 13:39:15.207316 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.207294    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54e0e73-1c73-430a-a65a-aa0c530d90d4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d54e0e73-1c73-430a-a65a-aa0c530d90d4" (UID: "d54e0e73-1c73-430a-a65a-aa0c530d90d4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:15.208867 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.208843    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d54e0e73-1c73-430a-a65a-aa0c530d90d4" (UID: "d54e0e73-1c73-430a-a65a-aa0c530d90d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:15.308302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.308241    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d54e0e73-1c73-430a-a65a-aa0c530d90d4-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:15.308302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:15.308268    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d54e0e73-1c73-430a-a65a-aa0c530d90d4-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:16.099072 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:16.099049    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"
Jul 02 13:39:16.099072 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:16.099066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c" event={"ID":"d54e0e73-1c73-430a-a65a-aa0c530d90d4","Type":"ContainerDied","Data":"823acdf8cc0970ced5e4f0e9762b3a07e3f3f9ca90fa84652f0429dd2a4e3229"}
Jul 02 13:39:16.099646 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:16.099119    2555 scope.go:117] "RemoveContainer" containerID="d2a736f4501430f1bed1ee0ccbe70dfcf33c1eeb62f67b2b75788a7c9842276a"
Jul 02 13:39:16.114385 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:16.114361    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"]
Jul 02 13:39:16.122208 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:16.119088    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-383ed-79cc5cfd9b-cjs9c"]
Jul 02 13:39:17.225849 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:17.225822    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" path="/var/lib/kubelet/pods/d54e0e73-1c73-430a-a65a-aa0c530d90d4/volumes"
Jul 02 13:39:22.973595 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:22.973561    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 02 13:39:24.769660 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.769624    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"]
Jul 02 13:39:24.770137 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.769887    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f" containerID="cri-o://44b95d92f82e7217244074768ec6c82cd8418c62051cfe3576c21f354bd3e554" gracePeriod=30
Jul 02 13:39:24.841626 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.841587    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"]
Jul 02 13:39:24.841929 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.841897    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" containerID="cri-o://bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962" gracePeriod=30
Jul 02 13:39:24.842166 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.841960    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kube-rbac-proxy" containerID="cri-o://50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c" gracePeriod=30
Jul 02 13:39:24.988406 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988368    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"]
Jul 02 13:39:24.988801 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988783    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kube-rbac-proxy"
Jul 02 13:39:24.988801 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988802    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kube-rbac-proxy"
Jul 02 13:39:24.988947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988825    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container"
Jul 02 13:39:24.988947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988833    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container"
Jul 02 13:39:24.988947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988852    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed"
Jul 02 13:39:24.988947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988862    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed"
Jul 02 13:39:24.988947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988947    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kube-rbac-proxy"
Jul 02 13:39:24.989188 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988963    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba78be6a-6f3b-419c-97dc-1c86c90721a1" containerName="kserve-container"
Jul 02 13:39:24.989188 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.988974    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54e0e73-1c73-430a-a65a-aa0c530d90d4" containerName="switch-graph-383ed"
Jul 02 13:39:24.992006 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.991988    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:24.994402 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.994380    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2bc9c-predictor-serving-cert\""
Jul 02 13:39:24.994527 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:24.994436    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2bc9c-kube-rbac-proxy-sar-config\""
Jul 02 13:39:25.001393 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.001371    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"]
Jul 02 13:39:25.091487 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.091450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c576c\" (UniqueName: \"kubernetes.io/projected/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-api-access-c576c\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.091734 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.091493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.091734 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.091600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.139898 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.139872    2555 generic.go:356] "Generic (PLEG): container finished" podID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerID="50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c" exitCode=2
Jul 02 13:39:25.140019 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.139947    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" event={"ID":"3ca3c6a8-3ce7-469a-bf23-76cea2611e36","Type":"ContainerDied","Data":"50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c"}
Jul 02 13:39:25.192964 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.192933    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c576c\" (UniqueName: \"kubernetes.io/projected/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-api-access-c576c\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.192964 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.192966    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.193114 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.193033    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.193169 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:25.193147    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-serving-cert: secret "error-404-isvc-2bc9c-predictor-serving-cert" not found
Jul 02 13:39:25.193249 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:25.193237    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls podName:aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc nodeName:}" failed. No retries permitted until 2026-07-02 13:39:25.693217025 +0000 UTC m=+1472.883546688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls") pod "error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" (UID: "aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc") : secret "error-404-isvc-2bc9c-predictor-serving-cert" not found
Jul 02 13:39:25.193659 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.193643    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.203726 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.203696    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c576c\" (UniqueName: \"kubernetes.io/projected/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-api-access-c576c\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.696664 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.696627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.699083 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.699061    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls\") pod \"error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") " pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:25.903033 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:25.902997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:26.041964 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:26.041941    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"]
Jul 02 13:39:26.044547 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:39:26.044522    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa940ca5_6df1_4e8f_b8b6_cb9b17811cfc.slice/crio-778c86dd3be07f861b8a9bc9907f757916937fc583bf2d98d51ab86e721a9cf3 WatchSource:0}: Error finding container 778c86dd3be07f861b8a9bc9907f757916937fc583bf2d98d51ab86e721a9cf3: Status 404 returned error can't find the container with id 778c86dd3be07f861b8a9bc9907f757916937fc583bf2d98d51ab86e721a9cf3
Jul 02 13:39:26.145408 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:26.145382    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" event={"ID":"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc","Type":"ContainerStarted","Data":"0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770"}
Jul 02 13:39:26.145505 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:26.145437    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" event={"ID":"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc","Type":"ContainerStarted","Data":"778c86dd3be07f861b8a9bc9907f757916937fc583bf2d98d51ab86e721a9cf3"}
Jul 02 13:39:26.780036 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:26.779993    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 02 13:39:26.786467 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:26.786432    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 02 13:39:27.052357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.052274    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:27.151001 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.150966    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" event={"ID":"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc","Type":"ContainerStarted","Data":"7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576"}
Jul 02 13:39:27.151174 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.151070    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:27.168955 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.168910    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podStartSLOduration=3.168895861 podStartE2EDuration="3.168895861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:39:27.167123498 +0000 UTC m=+1474.357453171" watchObservedRunningTime="2026-07-02 13:39:27.168895861 +0000 UTC m=+1474.359225596"
Jul 02 13:39:27.790387 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.790352    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:39:27.915202 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.914442    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-rbac-proxy-sar-config\") pod \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") "
Jul 02 13:39:27.915202 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.914526    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8nvs\" (UniqueName: \"kubernetes.io/projected/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-api-access-h8nvs\") pod \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") "
Jul 02 13:39:27.915202 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.914635    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-proxy-tls\") pod \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\" (UID: \"3ca3c6a8-3ce7-469a-bf23-76cea2611e36\") "
Jul 02 13:39:27.918090 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.917874    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ca3c6a8-3ce7-469a-bf23-76cea2611e36" (UID: "3ca3c6a8-3ce7-469a-bf23-76cea2611e36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:27.918590 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.918324    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ca3c6a8-3ce7-469a-bf23-76cea2611e36" (UID: "3ca3c6a8-3ce7-469a-bf23-76cea2611e36"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:27.921044 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:27.921014    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-api-access-h8nvs" (OuterVolumeSpecName: "kube-api-access-h8nvs") pod "3ca3c6a8-3ce7-469a-bf23-76cea2611e36" (UID: "3ca3c6a8-3ce7-469a-bf23-76cea2611e36"). InnerVolumeSpecName "kube-api-access-h8nvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:28.016359 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.016324    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:28.016359 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.016352    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nvs\" (UniqueName: \"kubernetes.io/projected/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-kube-api-access-h8nvs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:28.016359 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.016362    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ca3c6a8-3ce7-469a-bf23-76cea2611e36-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:28.157166 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.157129    2555 generic.go:356] "Generic (PLEG): container finished" podID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerID="bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962" exitCode=0
Jul 02 13:39:28.157584 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.157199    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"
Jul 02 13:39:28.157584 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.157212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" event={"ID":"3ca3c6a8-3ce7-469a-bf23-76cea2611e36","Type":"ContainerDied","Data":"bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962"}
Jul 02 13:39:28.157584 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.157296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x" event={"ID":"3ca3c6a8-3ce7-469a-bf23-76cea2611e36","Type":"ContainerDied","Data":"c69a1bd3134f79fe020ba904f3181c1a763a07de4f252e17a885b1faf1a5ed1e"}
Jul 02 13:39:28.157584 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.157326    2555 scope.go:117] "RemoveContainer" containerID="50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c"
Jul 02 13:39:28.157843 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.157753    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:28.159366 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.159339    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:39:28.166335 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.166317    2555 scope.go:117] "RemoveContainer" containerID="bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962"
Jul 02 13:39:28.173374 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.173356    2555 scope.go:117] "RemoveContainer" containerID="50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c"
Jul 02 13:39:28.173627 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:28.173609    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c\": container with ID starting with 50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c not found: ID does not exist" containerID="50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c"
Jul 02 13:39:28.173672 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.173635    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c"} err="failed to get container status \"50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c\": rpc error: code = NotFound desc = could not find container \"50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c\": container with ID starting with 50807f7adcd5f2bc259de10fdff233e795cd3d55db7b27194dad8cc6061d7c1c not found: ID does not exist"
Jul 02 13:39:28.173672 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.173651    2555 scope.go:117] "RemoveContainer" containerID="bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962"
Jul 02 13:39:28.173892 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:28.173875    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962\": container with ID starting with bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962 not found: ID does not exist" containerID="bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962"
Jul 02 13:39:28.173929 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.173898    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962"} err="failed to get container status \"bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962\": rpc error: code = NotFound desc = could not find container \"bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962\": container with ID starting with bf73ea3acf50a0d3ac4d4aaf0f9f13486d8b087ec70298647f8e3b6e486f3962 not found: ID does not exist"
Jul 02 13:39:28.182396 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.182373    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"]
Jul 02 13:39:28.186537 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:28.186516    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f369f-predictor-784754f556-n6j5x"]
Jul 02 13:39:29.162241 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:29.162209    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:39:29.224918 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:29.224889    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" path="/var/lib/kubelet/pods/3ca3c6a8-3ce7-469a-bf23-76cea2611e36/volumes"
Jul 02 13:39:32.052335 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:32.052300    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:32.974430 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:32.974405    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:39:34.166191 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:34.166164    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:39:34.166612 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:34.166587    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:39:37.052280 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:37.052241    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:37.052643 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:37.052339    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:39:42.052458 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:42.052416    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:44.166845 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:44.166807    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:39:45.232277 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.232239    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"]
Jul 02 13:39:45.232838 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.232722    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container"
Jul 02 13:39:45.232838 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.232741    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container"
Jul 02 13:39:45.232838 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.232756    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kube-rbac-proxy"
Jul 02 13:39:45.232838 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.232771    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kube-rbac-proxy"
Jul 02 13:39:45.233068 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.232869    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kserve-container"
Jul 02 13:39:45.233068 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.232911    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca3c6a8-3ce7-469a-bf23-76cea2611e36" containerName="kube-rbac-proxy"
Jul 02 13:39:45.237442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.237421    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.240766 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.240736    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-6c5e8-serving-cert\""
Jul 02 13:39:45.240897 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.240881    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-6c5e8-kube-rbac-proxy-sar-config\""
Jul 02 13:39:45.246369 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.246332    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"]
Jul 02 13:39:45.353568 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.353533    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-openshift-service-ca-bundle\") pod \"ensemble-graph-6c5e8-5fd6dd9659-55l9b\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") " pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.353756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.353608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-proxy-tls\") pod \"ensemble-graph-6c5e8-5fd6dd9659-55l9b\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") " pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.454365 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.454331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-openshift-service-ca-bundle\") pod \"ensemble-graph-6c5e8-5fd6dd9659-55l9b\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") " pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.454504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.454389    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-proxy-tls\") pod \"ensemble-graph-6c5e8-5fd6dd9659-55l9b\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") " pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.455039 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.455013    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-openshift-service-ca-bundle\") pod \"ensemble-graph-6c5e8-5fd6dd9659-55l9b\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") " pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.456767 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.456745    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-proxy-tls\") pod \"ensemble-graph-6c5e8-5fd6dd9659-55l9b\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") " pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.548617 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.548530    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:45.681504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:45.678983    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"]
Jul 02 13:39:45.682222 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:39:45.682192    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a776a68_e7f8_41ba_8639_a81c33ab1bd8.slice/crio-9355c2a9fbcf572821e30efb53d2dc57c9c878ce475d5c2d60cd1272c23b1f46 WatchSource:0}: Error finding container 9355c2a9fbcf572821e30efb53d2dc57c9c878ce475d5c2d60cd1272c23b1f46: Status 404 returned error can't find the container with id 9355c2a9fbcf572821e30efb53d2dc57c9c878ce475d5c2d60cd1272c23b1f46
Jul 02 13:39:46.232613 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:46.232582    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" event={"ID":"3a776a68-e7f8-41ba-8639-a81c33ab1bd8","Type":"ContainerStarted","Data":"792a597887d899469f41fa0c574741be7a5f468991d9e26acb66f31dc0d5bedd"}
Jul 02 13:39:46.233037 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:46.232631    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" event={"ID":"3a776a68-e7f8-41ba-8639-a81c33ab1bd8","Type":"ContainerStarted","Data":"9355c2a9fbcf572821e30efb53d2dc57c9c878ce475d5c2d60cd1272c23b1f46"}
Jul 02 13:39:46.233037 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:46.232733    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:46.248094 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:46.248047    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podStartSLOduration=1.248033116 podStartE2EDuration="1.248033116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:39:46.247023536 +0000 UTC m=+1493.437353208" watchObservedRunningTime="2026-07-02 13:39:46.248033116 +0000 UTC m=+1493.438362822"
Jul 02 13:39:47.052292 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:47.052256    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:52.051650 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:52.051608    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:52.242416 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:52.242389    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:39:53.265150 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:53.265126    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:39:53.266164 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:53.266144    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:39:53.270787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:53.270763    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:39:53.272336 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:53.272317    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:39:54.167526 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:54.167488    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:39:55.269626 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.269594    2555 generic.go:356] "Generic (PLEG): container finished" podID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerID="44b95d92f82e7217244074768ec6c82cd8418c62051cfe3576c21f354bd3e554" exitCode=0
Jul 02 13:39:55.270021 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.269653    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" event={"ID":"07fb52c1-e059-4a93-9aea-3468ebc711f6","Type":"ContainerDied","Data":"44b95d92f82e7217244074768ec6c82cd8418c62051cfe3576c21f354bd3e554"}
Jul 02 13:39:55.302633 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.302595    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"]
Jul 02 13:39:55.302870 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.302842    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8" containerID="cri-o://792a597887d899469f41fa0c574741be7a5f468991d9e26acb66f31dc0d5bedd" gracePeriod=30
Jul 02 13:39:55.378610 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.378574    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"]
Jul 02 13:39:55.378905 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.378879    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container" containerID="cri-o://0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a" gracePeriod=30
Jul 02 13:39:55.378995 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.378923    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kube-rbac-proxy" containerID="cri-o://f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3" gracePeriod=30
Jul 02 13:39:55.505213 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.505192    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:39:55.532887 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.532817    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"]
Jul 02 13:39:55.533358 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.533340    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f"
Jul 02 13:39:55.533399 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.533362    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f"
Jul 02 13:39:55.533457 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.533446    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" containerName="sequence-graph-f369f"
Jul 02 13:39:55.535297 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.535276    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fb52c1-e059-4a93-9aea-3468ebc711f6-openshift-service-ca-bundle\") pod \"07fb52c1-e059-4a93-9aea-3468ebc711f6\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") "
Jul 02 13:39:55.535395 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.535334    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fb52c1-e059-4a93-9aea-3468ebc711f6-proxy-tls\") pod \"07fb52c1-e059-4a93-9aea-3468ebc711f6\" (UID: \"07fb52c1-e059-4a93-9aea-3468ebc711f6\") "
Jul 02 13:39:55.535618 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.535591    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fb52c1-e059-4a93-9aea-3468ebc711f6-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "07fb52c1-e059-4a93-9aea-3468ebc711f6" (UID: "07fb52c1-e059-4a93-9aea-3468ebc711f6"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:55.537081 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.537062    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.537364 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.537326    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fb52c1-e059-4a93-9aea-3468ebc711f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07fb52c1-e059-4a93-9aea-3468ebc711f6" (UID: "07fb52c1-e059-4a93-9aea-3468ebc711f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:55.539913 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.539893    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e2bd2-predictor-serving-cert\""
Jul 02 13:39:55.539996 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.539921    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e2bd2-kube-rbac-proxy-sar-config\""
Jul 02 13:39:55.544545 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.544416    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"]
Jul 02 13:39:55.636194 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.636160    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.636341 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.636219    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e000058e-9278-448d-8993-fd666a913d9e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.636341 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.636290    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmszs\" (UniqueName: \"kubernetes.io/projected/e000058e-9278-448d-8993-fd666a913d9e-kube-api-access-dmszs\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.636590 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.636345    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07fb52c1-e059-4a93-9aea-3468ebc711f6-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:55.636590 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.636357    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07fb52c1-e059-4a93-9aea-3468ebc711f6-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:55.737340 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.737304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e000058e-9278-448d-8993-fd666a913d9e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.737536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.737349    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmszs\" (UniqueName: \"kubernetes.io/projected/e000058e-9278-448d-8993-fd666a913d9e-kube-api-access-dmszs\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.737536 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.737379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.737536 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:55.737494    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-serving-cert: secret "error-404-isvc-e2bd2-predictor-serving-cert" not found
Jul 02 13:39:55.737779 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:55.737562    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls podName:e000058e-9278-448d-8993-fd666a913d9e nodeName:}" failed. No retries permitted until 2026-07-02 13:39:56.237543938 +0000 UTC m=+1503.427873601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls") pod "error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" (UID: "e000058e-9278-448d-8993-fd666a913d9e") : secret "error-404-isvc-e2bd2-predictor-serving-cert" not found
Jul 02 13:39:55.738072 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.738049    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e000058e-9278-448d-8993-fd666a913d9e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:55.749204 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:55.749184    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmszs\" (UniqueName: \"kubernetes.io/projected/e000058e-9278-448d-8993-fd666a913d9e-kube-api-access-dmszs\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:56.243014 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.242980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:56.245375 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.245343    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls\") pod \"error-404-isvc-e2bd2-predictor-677686d8bb-mxljv\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") " pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:56.275222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.275194    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"
Jul 02 13:39:56.275222 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.275213    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z" event={"ID":"07fb52c1-e059-4a93-9aea-3468ebc711f6","Type":"ContainerDied","Data":"617b8b6d04ceb6604fbb21c1caf5df89043120dec72706e7b144a5643b0530b4"}
Jul 02 13:39:56.275637 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.275271    2555 scope.go:117] "RemoveContainer" containerID="44b95d92f82e7217244074768ec6c82cd8418c62051cfe3576c21f354bd3e554"
Jul 02 13:39:56.277192 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.277166    2555 generic.go:356] "Generic (PLEG): container finished" podID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerID="f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3" exitCode=2
Jul 02 13:39:56.277273 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.277203    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" event={"ID":"fcd366e1-f14f-4f7b-846c-987e5b4b5298","Type":"ContainerDied","Data":"f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3"}
Jul 02 13:39:56.296071 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.296051    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"]
Jul 02 13:39:56.299439 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.299420    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f369f-d9b69b6db-2jn9z"]
Jul 02 13:39:56.450298 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.450267    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:56.581566 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:56.581545    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"]
Jul 02 13:39:56.584354 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:39:56.584329    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode000058e_9278_448d_8993_fd666a913d9e.slice/crio-434d8ceeff4aa8589c7e4e26d2eddf673ee42d946485a2bae6ba57c61857b28d WatchSource:0}: Error finding container 434d8ceeff4aa8589c7e4e26d2eddf673ee42d946485a2bae6ba57c61857b28d: Status 404 returned error can't find the container with id 434d8ceeff4aa8589c7e4e26d2eddf673ee42d946485a2bae6ba57c61857b28d
Jul 02 13:39:57.225719 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.225681    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07fb52c1-e059-4a93-9aea-3468ebc711f6" path="/var/lib/kubelet/pods/07fb52c1-e059-4a93-9aea-3468ebc711f6/volumes"
Jul 02 13:39:57.240725 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.240678    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:39:57.291450 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.291411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" event={"ID":"e000058e-9278-448d-8993-fd666a913d9e","Type":"ContainerStarted","Data":"55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b"}
Jul 02 13:39:57.291931 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.291523    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:57.291931 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.291560    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:39:57.291931 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.291590    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" event={"ID":"e000058e-9278-448d-8993-fd666a913d9e","Type":"ContainerStarted","Data":"c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa"}
Jul 02 13:39:57.291931 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.291620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" event={"ID":"e000058e-9278-448d-8993-fd666a913d9e","Type":"ContainerStarted","Data":"434d8ceeff4aa8589c7e4e26d2eddf673ee42d946485a2bae6ba57c61857b28d"}
Jul 02 13:39:57.292981 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.292951    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:39:57.308608 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.308557    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podStartSLOduration=2.308541861 podStartE2EDuration="2.308541861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:39:57.307135692 +0000 UTC m=+1504.497465399" watchObservedRunningTime="2026-07-02 13:39:57.308541861 +0000 UTC m=+1504.498871531"
Jul 02 13:39:57.969871 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:57.969828    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 02 13:39:58.297420 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.297325    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:39:58.725300 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.725278    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:39:58.766511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.766488    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-rbac-proxy-sar-config\") pod \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") "
Jul 02 13:39:58.766628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.766529    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm4gq\" (UniqueName: \"kubernetes.io/projected/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-api-access-jm4gq\") pod \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") "
Jul 02 13:39:58.766628 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.766573    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcd366e1-f14f-4f7b-846c-987e5b4b5298-proxy-tls\") pod \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\" (UID: \"fcd366e1-f14f-4f7b-846c-987e5b4b5298\") "
Jul 02 13:39:58.766893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.766868    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcd366e1-f14f-4f7b-846c-987e5b4b5298" (UID: "fcd366e1-f14f-4f7b-846c-987e5b4b5298"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:39:58.768579 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.768547    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-api-access-jm4gq" (OuterVolumeSpecName: "kube-api-access-jm4gq") pod "fcd366e1-f14f-4f7b-846c-987e5b4b5298" (UID: "fcd366e1-f14f-4f7b-846c-987e5b4b5298"). InnerVolumeSpecName "kube-api-access-jm4gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:39:58.768668 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.768613    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd366e1-f14f-4f7b-846c-987e5b4b5298-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcd366e1-f14f-4f7b-846c-987e5b4b5298" (UID: "fcd366e1-f14f-4f7b-846c-987e5b4b5298"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:39:58.867267 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.867190    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:58.867267 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.867216    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4gq\" (UniqueName: \"kubernetes.io/projected/fcd366e1-f14f-4f7b-846c-987e5b4b5298-kube-api-access-jm4gq\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:58.867267 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:58.867229    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcd366e1-f14f-4f7b-846c-987e5b4b5298-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:39:59.303468 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.303435    2555 generic.go:356] "Generic (PLEG): container finished" podID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerID="0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a" exitCode=0
Jul 02 13:39:59.303912 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.303484    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"
Jul 02 13:39:59.303912 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.303519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" event={"ID":"fcd366e1-f14f-4f7b-846c-987e5b4b5298","Type":"ContainerDied","Data":"0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a"}
Jul 02 13:39:59.303912 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.303583    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd" event={"ID":"fcd366e1-f14f-4f7b-846c-987e5b4b5298","Type":"ContainerDied","Data":"8247aeaf7dc1fdfb4e6524b5520009e6ace47593652daccced6c7318802e3649"}
Jul 02 13:39:59.303912 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.303623    2555 scope.go:117] "RemoveContainer" containerID="f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3"
Jul 02 13:39:59.312152 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.312136    2555 scope.go:117] "RemoveContainer" containerID="0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a"
Jul 02 13:39:59.319452 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.319437    2555 scope.go:117] "RemoveContainer" containerID="f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3"
Jul 02 13:39:59.319735 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:59.319688    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3\": container with ID starting with f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3 not found: ID does not exist" containerID="f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3"
Jul 02 13:39:59.319813 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.319733    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3"} err="failed to get container status \"f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3\": rpc error: code = NotFound desc = could not find container \"f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3\": container with ID starting with f258a2344f8da944ef4d244ea4604bcc66b20fe7f8ac7737c860b0e16c098df3 not found: ID does not exist"
Jul 02 13:39:59.319813 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.319750    2555 scope.go:117] "RemoveContainer" containerID="0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a"
Jul 02 13:39:59.320003 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:39:59.319985    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a\": container with ID starting with 0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a not found: ID does not exist" containerID="0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a"
Jul 02 13:39:59.320049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.320007    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a"} err="failed to get container status \"0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a\": rpc error: code = NotFound desc = could not find container \"0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a\": container with ID starting with 0571cb07703f6662121ee87eaceba1edb6e5a06389472ba1dbe9b8cd94ea9d7a not found: ID does not exist"
Jul 02 13:39:59.320443 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.320427    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"]
Jul 02 13:39:59.323682 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:39:59.323662    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c5e8-predictor-bbf5dc8fd-5h8zd"]
Jul 02 13:40:01.225629 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:01.225588    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" path="/var/lib/kubelet/pods/fcd366e1-f14f-4f7b-846c-987e5b4b5298/volumes"
Jul 02 13:40:02.241085 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:02.241046    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:03.303770 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:03.303740    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:40:03.304335 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:03.304307    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:40:04.166565 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:04.166524    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 02 13:40:07.241391 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:07.241347    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:07.241802 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:07.241440    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:40:12.241260 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:12.241222    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:13.305167 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:13.305123    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:40:14.167755 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:14.167728    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:40:17.241075 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:17.241036    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:22.241137 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:22.241099    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:23.305144 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:23.305104    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:40:24.916206 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.916166    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"]
Jul 02 13:40:24.916790 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.916741    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container"
Jul 02 13:40:24.916790 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.916762    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container"
Jul 02 13:40:24.916790 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.916782    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kube-rbac-proxy"
Jul 02 13:40:24.916790 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.916791    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kube-rbac-proxy"
Jul 02 13:40:24.917021 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.916874    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kube-rbac-proxy"
Jul 02 13:40:24.917021 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.916885    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcd366e1-f14f-4f7b-846c-987e5b4b5298" containerName="kserve-container"
Jul 02 13:40:24.919682 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.919661    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:24.922105 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.922084    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2bc9c-serving-cert\""
Jul 02 13:40:24.922220 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.922118    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2bc9c-kube-rbac-proxy-sar-config\""
Jul 02 13:40:24.928106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.928085    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"]
Jul 02 13:40:24.973603 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.973579    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc5fffe-c874-4979-a16b-307f7b3e2f53-proxy-tls\") pod \"sequence-graph-2bc9c-6db49dd9b7-dfmqj\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") " pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:24.973760 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:24.973625    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc5fffe-c874-4979-a16b-307f7b3e2f53-openshift-service-ca-bundle\") pod \"sequence-graph-2bc9c-6db49dd9b7-dfmqj\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") " pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:25.074201 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.074167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc5fffe-c874-4979-a16b-307f7b3e2f53-openshift-service-ca-bundle\") pod \"sequence-graph-2bc9c-6db49dd9b7-dfmqj\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") " pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:25.074399 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.074333    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc5fffe-c874-4979-a16b-307f7b3e2f53-proxy-tls\") pod \"sequence-graph-2bc9c-6db49dd9b7-dfmqj\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") " pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:25.074908 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.074880    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc5fffe-c874-4979-a16b-307f7b3e2f53-openshift-service-ca-bundle\") pod \"sequence-graph-2bc9c-6db49dd9b7-dfmqj\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") " pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:25.076770 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.076746    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc5fffe-c874-4979-a16b-307f7b3e2f53-proxy-tls\") pod \"sequence-graph-2bc9c-6db49dd9b7-dfmqj\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") " pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:25.231429 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.231359    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:25.364778 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.364700    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"]
Jul 02 13:40:25.377619 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:40:25.377593    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc5fffe_c874_4979_a16b_307f7b3e2f53.slice/crio-8ff3a5edd0153794de6260f7ff6978108f12247f08c1f39917b4f3d1621932fd WatchSource:0}: Error finding container 8ff3a5edd0153794de6260f7ff6978108f12247f08c1f39917b4f3d1621932fd: Status 404 returned error can't find the container with id 8ff3a5edd0153794de6260f7ff6978108f12247f08c1f39917b4f3d1621932fd
Jul 02 13:40:25.428398 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.428372    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" event={"ID":"8dc5fffe-c874-4979-a16b-307f7b3e2f53","Type":"ContainerStarted","Data":"8ff3a5edd0153794de6260f7ff6978108f12247f08c1f39917b4f3d1621932fd"}
Jul 02 13:40:25.429636 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.429611    2555 generic.go:356] "Generic (PLEG): container finished" podID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerID="792a597887d899469f41fa0c574741be7a5f468991d9e26acb66f31dc0d5bedd" exitCode=0
Jul 02 13:40:25.429747 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.429648    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" event={"ID":"3a776a68-e7f8-41ba-8639-a81c33ab1bd8","Type":"ContainerDied","Data":"792a597887d899469f41fa0c574741be7a5f468991d9e26acb66f31dc0d5bedd"}
Jul 02 13:40:25.938042 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:25.938021    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:40:26.083640 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.083615    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-proxy-tls\") pod \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") "
Jul 02 13:40:26.083789 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.083654    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-openshift-service-ca-bundle\") pod \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\" (UID: \"3a776a68-e7f8-41ba-8639-a81c33ab1bd8\") "
Jul 02 13:40:26.084063 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.084036    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3a776a68-e7f8-41ba-8639-a81c33ab1bd8" (UID: "3a776a68-e7f8-41ba-8639-a81c33ab1bd8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:40:26.085872 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.085851    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a776a68-e7f8-41ba-8639-a81c33ab1bd8" (UID: "3a776a68-e7f8-41ba-8639-a81c33ab1bd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:40:26.184592 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.184555    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:40:26.184592 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.184594    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a776a68-e7f8-41ba-8639-a81c33ab1bd8-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:40:26.434453 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.434379    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"
Jul 02 13:40:26.434453 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.434431    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b" event={"ID":"3a776a68-e7f8-41ba-8639-a81c33ab1bd8","Type":"ContainerDied","Data":"9355c2a9fbcf572821e30efb53d2dc57c9c878ce475d5c2d60cd1272c23b1f46"}
Jul 02 13:40:26.434661 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.434489    2555 scope.go:117] "RemoveContainer" containerID="792a597887d899469f41fa0c574741be7a5f468991d9e26acb66f31dc0d5bedd"
Jul 02 13:40:26.435980 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.435952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" event={"ID":"8dc5fffe-c874-4979-a16b-307f7b3e2f53","Type":"ContainerStarted","Data":"3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b"}
Jul 02 13:40:26.436119 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.436002    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:26.457117 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.457078    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podStartSLOduration=2.457066345 podStartE2EDuration="2.457066345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:40:26.455227866 +0000 UTC m=+1533.645557531" watchObservedRunningTime="2026-07-02 13:40:26.457066345 +0000 UTC m=+1533.647396016"
Jul 02 13:40:26.467680 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.467660    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"]
Jul 02 13:40:26.471332 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:26.471314    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-6c5e8-5fd6dd9659-55l9b"]
Jul 02 13:40:27.225652 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:27.225623    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" path="/var/lib/kubelet/pods/3a776a68-e7f8-41ba-8639-a81c33ab1bd8/volumes"
Jul 02 13:40:32.448336 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:32.448306    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:33.304258 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:33.304220    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 02 13:40:34.999137 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:34.999104    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"]
Jul 02 13:40:34.999547 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:34.999345    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c" containerID="cri-o://3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b" gracePeriod=30
Jul 02 13:40:35.080670 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.080634    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"]
Jul 02 13:40:35.081064 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.080902    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container" containerID="cri-o://0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770" gracePeriod=30
Jul 02 13:40:35.081164 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.081057    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kube-rbac-proxy" containerID="cri-o://7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576" gracePeriod=30
Jul 02 13:40:35.212291 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.212261    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"]
Jul 02 13:40:35.212587 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.212575    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8"
Jul 02 13:40:35.212627 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.212588    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8"
Jul 02 13:40:35.212665 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.212652    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a776a68-e7f8-41ba-8639-a81c33ab1bd8" containerName="ensemble-graph-6c5e8"
Jul 02 13:40:35.215615 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.215600    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.218177 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.218152    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7b15c-predictor-serving-cert\""
Jul 02 13:40:35.218283 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.218161    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7b15c-kube-rbac-proxy-sar-config\""
Jul 02 13:40:35.227681 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.227661    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"]
Jul 02 13:40:35.360842 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.360805    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-proxy-tls\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.361003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.360857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljq74\" (UniqueName: \"kubernetes.io/projected/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-api-access-ljq74\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.361003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.360910    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.461798 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.461753    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.461971 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.461855    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-proxy-tls\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.461971 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.461878    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljq74\" (UniqueName: \"kubernetes.io/projected/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-api-access-ljq74\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.462494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.462473    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.464286 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.464267    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-proxy-tls\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.469104 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.469087    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljq74\" (UniqueName: \"kubernetes.io/projected/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-api-access-ljq74\") pod \"error-404-isvc-7b15c-predictor-6c577469b4-7vqmd\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") " pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.478300 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.478275    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerID="7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576" exitCode=2
Jul 02 13:40:35.478421 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.478334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" event={"ID":"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc","Type":"ContainerDied","Data":"7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576"}
Jul 02 13:40:35.526490 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.526460    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:35.654662 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:35.654632    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"]
Jul 02 13:40:35.656839 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:40:35.656812    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a269e0_b1a9_40d8_b7a7_c4bdce124a68.slice/crio-10a9c1f840400ca23ecd7f76e300bc160c1fb3514cbb633dc295494f48e2890e WatchSource:0}: Error finding container 10a9c1f840400ca23ecd7f76e300bc160c1fb3514cbb633dc295494f48e2890e: Status 404 returned error can't find the container with id 10a9c1f840400ca23ecd7f76e300bc160c1fb3514cbb633dc295494f48e2890e
Jul 02 13:40:36.486811 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:36.486769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" event={"ID":"70a269e0-b1a9-40d8-b7a7-c4bdce124a68","Type":"ContainerStarted","Data":"9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31"}
Jul 02 13:40:36.487297 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:36.486879    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:36.487297 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:36.486906    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" event={"ID":"70a269e0-b1a9-40d8-b7a7-c4bdce124a68","Type":"ContainerStarted","Data":"687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed"}
Jul 02 13:40:36.487297 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:36.486925    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" event={"ID":"70a269e0-b1a9-40d8-b7a7-c4bdce124a68","Type":"ContainerStarted","Data":"10a9c1f840400ca23ecd7f76e300bc160c1fb3514cbb633dc295494f48e2890e"}
Jul 02 13:40:36.503778 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:36.503732    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podStartSLOduration=1.503695684 podStartE2EDuration="1.503695684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:40:36.502605724 +0000 UTC m=+1543.692935396" watchObservedRunningTime="2026-07-02 13:40:36.503695684 +0000 UTC m=+1543.694025354"
Jul 02 13:40:37.445362 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:37.445324    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:37.493571 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:37.493529    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:37.495214 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:37.495185    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 13:40:38.328944 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.328917    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:40:38.490503 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.490424    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls\") pod \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") "
Jul 02 13:40:38.490503 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.490479    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c576c\" (UniqueName: \"kubernetes.io/projected/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-api-access-c576c\") pod \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") "
Jul 02 13:40:38.490718 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.490530    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-rbac-proxy-sar-config\") pod \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\" (UID: \"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc\") "
Jul 02 13:40:38.491008 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.490969    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" (UID: "aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:40:38.492589 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.492561    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" (UID: "aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:40:38.492733 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.492640    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-api-access-c576c" (OuterVolumeSpecName: "kube-api-access-c576c") pod "aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" (UID: "aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc"). InnerVolumeSpecName "kube-api-access-c576c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:40:38.500176 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.500153    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerID="0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770" exitCode=0
Jul 02 13:40:38.500494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.500210    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"
Jul 02 13:40:38.500494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.500232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" event={"ID":"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc","Type":"ContainerDied","Data":"0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770"}
Jul 02 13:40:38.500494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.500297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9" event={"ID":"aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc","Type":"ContainerDied","Data":"778c86dd3be07f861b8a9bc9907f757916937fc583bf2d98d51ab86e721a9cf3"}
Jul 02 13:40:38.500494 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.500325    2555 scope.go:117] "RemoveContainer" containerID="7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576"
Jul 02 13:40:38.500974 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.500947    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 13:40:38.509032 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.509017    2555 scope.go:117] "RemoveContainer" containerID="0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770"
Jul 02 13:40:38.515980 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.515964    2555 scope.go:117] "RemoveContainer" containerID="7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576"
Jul 02 13:40:38.516210 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:40:38.516194    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576\": container with ID starting with 7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576 not found: ID does not exist" containerID="7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576"
Jul 02 13:40:38.516254 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.516216    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576"} err="failed to get container status \"7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576\": rpc error: code = NotFound desc = could not find container \"7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576\": container with ID starting with 7cf79bfb9102cb4f09c0d5a8b559686a532c067f9f94b904f89490e467184576 not found: ID does not exist"
Jul 02 13:40:38.516254 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.516231    2555 scope.go:117] "RemoveContainer" containerID="0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770"
Jul 02 13:40:38.516453 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:40:38.516438    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770\": container with ID starting with 0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770 not found: ID does not exist" containerID="0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770"
Jul 02 13:40:38.516502 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.516457    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770"} err="failed to get container status \"0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770\": rpc error: code = NotFound desc = could not find container \"0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770\": container with ID starting with 0cc8c2cc4c413c1194923d23f96b54f555aa268011487b20335c60d408bec770 not found: ID does not exist"
Jul 02 13:40:38.520555 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.520536    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"]
Jul 02 13:40:38.526214 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.526195    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2bc9c-predictor-54f54f9cbb-qrbv9"]
Jul 02 13:40:38.591213 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.591185    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:40:38.591213 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.591211    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c576c\" (UniqueName: \"kubernetes.io/projected/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-api-access-c576c\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:40:38.591411 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:38.591225    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:40:39.225579 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:39.225547    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" path="/var/lib/kubelet/pods/aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc/volumes"
Jul 02 13:40:42.445672 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:42.445640    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:43.304848 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:43.304822    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:40:43.506042 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:43.506015    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:40:43.506657 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:43.506626    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 13:40:47.445760 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:47.445714    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:47.446220 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:47.445829    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:40:52.445674 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:52.445636    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:53.506941 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:53.506902    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 13:40:55.499891 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.499861    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"]
Jul 02 13:40:55.500275 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.500176    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kube-rbac-proxy"
Jul 02 13:40:55.500275 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.500187    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kube-rbac-proxy"
Jul 02 13:40:55.500275 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.500202    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container"
Jul 02 13:40:55.500275 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.500208    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container"
Jul 02 13:40:55.500511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.500283    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kube-rbac-proxy"
Jul 02 13:40:55.500511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.500293    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa940ca5-6df1-4e8f-b8b6-cb9b17811cfc" containerName="kserve-container"
Jul 02 13:40:55.504734 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.504715    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:55.507142 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.507112    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e2bd2-kube-rbac-proxy-sar-config\""
Jul 02 13:40:55.507142 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.507126    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-e2bd2-serving-cert\""
Jul 02 13:40:55.510404 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.510347    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"]
Jul 02 13:40:55.524074 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.524054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d58bb1-400e-401d-b130-f384f4fddf79-openshift-service-ca-bundle\") pod \"ensemble-graph-e2bd2-fc854767f-spdpt\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") " pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:55.524164 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.524086    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls\") pod \"ensemble-graph-e2bd2-fc854767f-spdpt\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") " pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:55.625268 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.625237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d58bb1-400e-401d-b130-f384f4fddf79-openshift-service-ca-bundle\") pod \"ensemble-graph-e2bd2-fc854767f-spdpt\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") " pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:55.625414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.625276    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls\") pod \"ensemble-graph-e2bd2-fc854767f-spdpt\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") " pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:55.625463 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:40:55.625417    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-e2bd2-serving-cert: secret "ensemble-graph-e2bd2-serving-cert" not found
Jul 02 13:40:55.625510 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:40:55.625498    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls podName:b9d58bb1-400e-401d-b130-f384f4fddf79 nodeName:}" failed. No retries permitted until 2026-07-02 13:40:56.125476386 +0000 UTC m=+1563.315806056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls") pod "ensemble-graph-e2bd2-fc854767f-spdpt" (UID: "b9d58bb1-400e-401d-b130-f384f4fddf79") : secret "ensemble-graph-e2bd2-serving-cert" not found
Jul 02 13:40:55.625971 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:55.625953    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d58bb1-400e-401d-b130-f384f4fddf79-openshift-service-ca-bundle\") pod \"ensemble-graph-e2bd2-fc854767f-spdpt\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") " pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:56.128375 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:56.128347    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls\") pod \"ensemble-graph-e2bd2-fc854767f-spdpt\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") " pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:56.130698 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:56.130674    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls\") pod \"ensemble-graph-e2bd2-fc854767f-spdpt\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") " pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:56.415481 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:56.415407    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:56.541875 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:56.541843    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"]
Jul 02 13:40:56.544914 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:40:56.544889    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d58bb1_400e_401d_b130_f384f4fddf79.slice/crio-036ccc5cd8a104e2bb129a53d546a42d77a78dfefa1b23de3d5d7e4f2c06b80b WatchSource:0}: Error finding container 036ccc5cd8a104e2bb129a53d546a42d77a78dfefa1b23de3d5d7e4f2c06b80b: Status 404 returned error can't find the container with id 036ccc5cd8a104e2bb129a53d546a42d77a78dfefa1b23de3d5d7e4f2c06b80b
Jul 02 13:40:56.579172 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:56.579148    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" event={"ID":"b9d58bb1-400e-401d-b130-f384f4fddf79","Type":"ContainerStarted","Data":"036ccc5cd8a104e2bb129a53d546a42d77a78dfefa1b23de3d5d7e4f2c06b80b"}
Jul 02 13:40:57.445174 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:57.445136    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:40:57.585265 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:57.585231    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" event={"ID":"b9d58bb1-400e-401d-b130-f384f4fddf79","Type":"ContainerStarted","Data":"6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a"}
Jul 02 13:40:57.585676 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:57.585379    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:40:57.601174 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:40:57.601130    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podStartSLOduration=2.601118467 podStartE2EDuration="2.601118467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:40:57.599453955 +0000 UTC m=+1564.789783638" watchObservedRunningTime="2026-07-02 13:40:57.601118467 +0000 UTC m=+1564.791448137"
Jul 02 13:41:02.445661 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:02.445620    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:41:03.507381 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:03.507345    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 13:41:03.597010 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:03.596985    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:41:05.135388 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.135364    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:41:05.198633 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.198611    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc5fffe-c874-4979-a16b-307f7b3e2f53-proxy-tls\") pod \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") "
Jul 02 13:41:05.198812 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.198650    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc5fffe-c874-4979-a16b-307f7b3e2f53-openshift-service-ca-bundle\") pod \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\" (UID: \"8dc5fffe-c874-4979-a16b-307f7b3e2f53\") "
Jul 02 13:41:05.199013 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.198989    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc5fffe-c874-4979-a16b-307f7b3e2f53-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8dc5fffe-c874-4979-a16b-307f7b3e2f53" (UID: "8dc5fffe-c874-4979-a16b-307f7b3e2f53"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:41:05.200580 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.200561    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc5fffe-c874-4979-a16b-307f7b3e2f53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8dc5fffe-c874-4979-a16b-307f7b3e2f53" (UID: "8dc5fffe-c874-4979-a16b-307f7b3e2f53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:41:05.299357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.299297    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc5fffe-c874-4979-a16b-307f7b3e2f53-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:41:05.299357 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.299318    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dc5fffe-c874-4979-a16b-307f7b3e2f53-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:41:05.624220 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.624191    2555 generic.go:356] "Generic (PLEG): container finished" podID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerID="3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b" exitCode=0
Jul 02 13:41:05.624377 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.624235    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"
Jul 02 13:41:05.624377 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.624255    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" event={"ID":"8dc5fffe-c874-4979-a16b-307f7b3e2f53","Type":"ContainerDied","Data":"3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b"}
Jul 02 13:41:05.624377 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.624296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj" event={"ID":"8dc5fffe-c874-4979-a16b-307f7b3e2f53","Type":"ContainerDied","Data":"8ff3a5edd0153794de6260f7ff6978108f12247f08c1f39917b4f3d1621932fd"}
Jul 02 13:41:05.624377 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.624316    2555 scope.go:117] "RemoveContainer" containerID="3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b"
Jul 02 13:41:05.632887 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.632855    2555 scope.go:117] "RemoveContainer" containerID="3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b"
Jul 02 13:41:05.637313 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:41:05.637289    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b\": container with ID starting with 3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b not found: ID does not exist" containerID="3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b"
Jul 02 13:41:05.637415 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.637321    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b"} err="failed to get container status \"3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b\": rpc error: code = NotFound desc = could not find container \"3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b\": container with ID starting with 3fa4351ccaa8dc97a095e8dfc127ecc3443c3b291d8c1a5ee74d4289717c6c1b not found: ID does not exist"
Jul 02 13:41:05.641023 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.641002    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"]
Jul 02 13:41:05.644302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:05.644279    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2bc9c-6db49dd9b7-dfmqj"]
Jul 02 13:41:07.225219 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:07.225188    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" path="/var/lib/kubelet/pods/8dc5fffe-c874-4979-a16b-307f7b3e2f53/volumes"
Jul 02 13:41:13.506863 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:13.506828    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 02 13:41:23.507420 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:23.507390    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:41:35.189439 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.189406    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"]
Jul 02 13:41:35.190120 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.189771    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c"
Jul 02 13:41:35.190120 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.189788    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c"
Jul 02 13:41:35.190120 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.189912    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc5fffe-c874-4979-a16b-307f7b3e2f53" containerName="sequence-graph-2bc9c"
Jul 02 13:41:35.192934 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.192909    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.195531 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.195508    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-7b15c-serving-cert\""
Jul 02 13:41:35.195650 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.195508    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-7b15c-kube-rbac-proxy-sar-config\""
Jul 02 13:41:35.204813 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.204789    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"]
Jul 02 13:41:35.323765 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.323728    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3faf2de-97cd-44c2-9534-2c6d935c0447-proxy-tls\") pod \"sequence-graph-7b15c-657959bb75-qgx8p\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") " pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.323927 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.323806    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3faf2de-97cd-44c2-9534-2c6d935c0447-openshift-service-ca-bundle\") pod \"sequence-graph-7b15c-657959bb75-qgx8p\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") " pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.425216 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.425180    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3faf2de-97cd-44c2-9534-2c6d935c0447-openshift-service-ca-bundle\") pod \"sequence-graph-7b15c-657959bb75-qgx8p\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") " pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.425386 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.425245    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3faf2de-97cd-44c2-9534-2c6d935c0447-proxy-tls\") pod \"sequence-graph-7b15c-657959bb75-qgx8p\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") " pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.425961 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.425933    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3faf2de-97cd-44c2-9534-2c6d935c0447-openshift-service-ca-bundle\") pod \"sequence-graph-7b15c-657959bb75-qgx8p\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") " pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.427584 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.427558    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3faf2de-97cd-44c2-9534-2c6d935c0447-proxy-tls\") pod \"sequence-graph-7b15c-657959bb75-qgx8p\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") " pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.503842 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.503760    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.629201 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.629144    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"]
Jul 02 13:41:35.631403 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:41:35.631379    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3faf2de_97cd_44c2_9534_2c6d935c0447.slice/crio-f3b5d81b1714f929b89d875918803e86dd2430c04d22cf0afe39328fad804630 WatchSource:0}: Error finding container f3b5d81b1714f929b89d875918803e86dd2430c04d22cf0afe39328fad804630: Status 404 returned error can't find the container with id f3b5d81b1714f929b89d875918803e86dd2430c04d22cf0afe39328fad804630
Jul 02 13:41:35.752627 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.752587    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" event={"ID":"f3faf2de-97cd-44c2-9534-2c6d935c0447","Type":"ContainerStarted","Data":"173a3d65eab0b8bf4245770dea298ca6ed31146e9a1fd5b2a25f7f961f6dfe5a"}
Jul 02 13:41:35.752791 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.752651    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" event={"ID":"f3faf2de-97cd-44c2-9534-2c6d935c0447","Type":"ContainerStarted","Data":"f3b5d81b1714f929b89d875918803e86dd2430c04d22cf0afe39328fad804630"}
Jul 02 13:41:35.752791 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.752690    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:41:35.768542 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:35.768460    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podStartSLOduration=0.768444988 podStartE2EDuration="768.444988ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:41:35.767767236 +0000 UTC m=+1602.958096907" watchObservedRunningTime="2026-07-02 13:41:35.768444988 +0000 UTC m=+1602.958774659"
Jul 02 13:41:41.763425 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:41:41.763391    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:44:53.293770 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:44:53.293741    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:44:53.295729 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:44:53.295693    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:44:53.298133 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:44:53.298114    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:44:53.300498 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:44:53.300482    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:49:15.246428 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.246333    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"]
Jul 02 13:49:15.248959 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.246588    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2" containerID="cri-o://6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a" gracePeriod=30
Jul 02 13:49:15.477870 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.477838    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"]
Jul 02 13:49:15.481207 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.481179    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.484075 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.484051    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9633b-kube-rbac-proxy-sar-config\""
Jul 02 13:49:15.484075 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.484051    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9633b-predictor-serving-cert\""
Jul 02 13:49:15.489838 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.489810    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"]
Jul 02 13:49:15.519747 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.519660    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4cbt\" (UniqueName: \"kubernetes.io/projected/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-api-access-s4cbt\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.519747 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.519727    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df728f4-3c51-46d9-bcf9-40f3da845c0a-proxy-tls\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.519947 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.519827    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.524101 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.524073    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"]
Jul 02 13:49:15.524375 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.524351    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kube-rbac-proxy" containerID="cri-o://55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b" gracePeriod=30
Jul 02 13:49:15.524375 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.524341    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container" containerID="cri-o://c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa" gracePeriod=30
Jul 02 13:49:15.620596 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.620570    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4cbt\" (UniqueName: \"kubernetes.io/projected/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-api-access-s4cbt\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.620773 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.620606    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df728f4-3c51-46d9-bcf9-40f3da845c0a-proxy-tls\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.620773 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.620637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.621347 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.621325    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.623112 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.623094    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df728f4-3c51-46d9-bcf9-40f3da845c0a-proxy-tls\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.627869 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.627847    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4cbt\" (UniqueName: \"kubernetes.io/projected/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-api-access-s4cbt\") pod \"error-404-isvc-9633b-predictor-6db75d69f8-hx5s5\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") " pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.740629 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.740598    2555 generic.go:356] "Generic (PLEG): container finished" podID="e000058e-9278-448d-8993-fd666a913d9e" containerID="55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b" exitCode=2
Jul 02 13:49:15.740805 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.740663    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" event={"ID":"e000058e-9278-448d-8993-fd666a913d9e","Type":"ContainerDied","Data":"55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b"}
Jul 02 13:49:15.792917 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.792847    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:15.923799 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:15.923722    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"]
Jul 02 13:49:15.926064 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:49:15.926034    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df728f4_3c51_46d9_bcf9_40f3da845c0a.slice/crio-3a0f4687d36e2184a21707d166c4289c7ce0b0f1c771f0a242b422188e88c622 WatchSource:0}: Error finding container 3a0f4687d36e2184a21707d166c4289c7ce0b0f1c771f0a242b422188e88c622: Status 404 returned error can't find the container with id 3a0f4687d36e2184a21707d166c4289c7ce0b0f1c771f0a242b422188e88c622
Jul 02 13:49:16.745970 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:16.745939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" event={"ID":"2df728f4-3c51-46d9-bcf9-40f3da845c0a","Type":"ContainerStarted","Data":"d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378"}
Jul 02 13:49:16.746442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:16.746009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:16.746442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:16.746040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" event={"ID":"2df728f4-3c51-46d9-bcf9-40f3da845c0a","Type":"ContainerStarted","Data":"5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b"}
Jul 02 13:49:16.746442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:16.746071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" event={"ID":"2df728f4-3c51-46d9-bcf9-40f3da845c0a","Type":"ContainerStarted","Data":"3a0f4687d36e2184a21707d166c4289c7ce0b0f1c771f0a242b422188e88c622"}
Jul 02 13:49:16.746442 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:16.746104    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:16.747512 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:16.747491    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 02 13:49:16.763087 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:16.763046    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podStartSLOduration=1.763033097 podStartE2EDuration="1.763033097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:16.761007073 +0000 UTC m=+2063.951336765" watchObservedRunningTime="2026-07-02 13:49:16.763033097 +0000 UTC m=+2063.953362828"
Jul 02 13:49:17.750616 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:17.750579    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 02 13:49:18.298419 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.298377    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 02 13:49:18.582923 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.582900    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:49:18.595269 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.595238    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:18.642474 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.642444    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls\") pod \"e000058e-9278-448d-8993-fd666a913d9e\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") "
Jul 02 13:49:18.642636 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.642523    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e000058e-9278-448d-8993-fd666a913d9e-kube-rbac-proxy-sar-config\") pod \"e000058e-9278-448d-8993-fd666a913d9e\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") "
Jul 02 13:49:18.642697 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.642678    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmszs\" (UniqueName: \"kubernetes.io/projected/e000058e-9278-448d-8993-fd666a913d9e-kube-api-access-dmszs\") pod \"e000058e-9278-448d-8993-fd666a913d9e\" (UID: \"e000058e-9278-448d-8993-fd666a913d9e\") "
Jul 02 13:49:18.642835 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.642815    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e000058e-9278-448d-8993-fd666a913d9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e000058e-9278-448d-8993-fd666a913d9e" (UID: "e000058e-9278-448d-8993-fd666a913d9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:49:18.642936 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.642923    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e000058e-9278-448d-8993-fd666a913d9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:18.644496 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.644476    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e000058e-9278-448d-8993-fd666a913d9e" (UID: "e000058e-9278-448d-8993-fd666a913d9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:49:18.644693 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.644672    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e000058e-9278-448d-8993-fd666a913d9e-kube-api-access-dmszs" (OuterVolumeSpecName: "kube-api-access-dmszs") pod "e000058e-9278-448d-8993-fd666a913d9e" (UID: "e000058e-9278-448d-8993-fd666a913d9e"). InnerVolumeSpecName "kube-api-access-dmszs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:49:18.743569 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.743534    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e000058e-9278-448d-8993-fd666a913d9e-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:18.743569 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.743566    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmszs\" (UniqueName: \"kubernetes.io/projected/e000058e-9278-448d-8993-fd666a913d9e-kube-api-access-dmszs\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:18.755699 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.755666    2555 generic.go:356] "Generic (PLEG): container finished" podID="e000058e-9278-448d-8993-fd666a913d9e" containerID="c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa" exitCode=0
Jul 02 13:49:18.756049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.755738    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"
Jul 02 13:49:18.756049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.755734    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" event={"ID":"e000058e-9278-448d-8993-fd666a913d9e","Type":"ContainerDied","Data":"c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa"}
Jul 02 13:49:18.756049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.755795    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv" event={"ID":"e000058e-9278-448d-8993-fd666a913d9e","Type":"ContainerDied","Data":"434d8ceeff4aa8589c7e4e26d2eddf673ee42d946485a2bae6ba57c61857b28d"}
Jul 02 13:49:18.756049 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.755822    2555 scope.go:117] "RemoveContainer" containerID="55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b"
Jul 02 13:49:18.764873 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.764856    2555 scope.go:117] "RemoveContainer" containerID="c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa"
Jul 02 13:49:18.772170 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.772153    2555 scope.go:117] "RemoveContainer" containerID="55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b"
Jul 02 13:49:18.772400 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:49:18.772383    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b\": container with ID starting with 55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b not found: ID does not exist" containerID="55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b"
Jul 02 13:49:18.772456 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.772408    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b"} err="failed to get container status \"55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b\": rpc error: code = NotFound desc = could not find container \"55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b\": container with ID starting with 55aae4afb517a3b707433be6911a6523a61ef434802eb6c25a7e295eb9fadd6b not found: ID does not exist"
Jul 02 13:49:18.772456 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.772427    2555 scope.go:117] "RemoveContainer" containerID="c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa"
Jul 02 13:49:18.772610 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:49:18.772596    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa\": container with ID starting with c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa not found: ID does not exist" containerID="c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa"
Jul 02 13:49:18.772652 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.772614    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa"} err="failed to get container status \"c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa\": rpc error: code = NotFound desc = could not find container \"c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa\": container with ID starting with c99c78a8f0a4b2c5a0a2375e50ee07ae52d23fb62b5bb81f948af59a370a8afa not found: ID does not exist"
Jul 02 13:49:18.777996 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.777976    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"]
Jul 02 13:49:18.780489 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:18.780467    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e2bd2-predictor-677686d8bb-mxljv"]
Jul 02 13:49:19.226331 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:19.226300    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e000058e-9278-448d-8993-fd666a913d9e" path="/var/lib/kubelet/pods/e000058e-9278-448d-8993-fd666a913d9e/volumes"
Jul 02 13:49:22.755137 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:22.755106    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:49:22.755522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:22.755493    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 02 13:49:23.594958 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:23.594922    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:28.595022 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:28.594983    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:28.595425 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:28.595086    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:49:32.755931 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:32.755892    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 02 13:49:33.594646 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:33.594605    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:38.594315 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:38.594277    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:42.755664 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:42.755618    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 02 13:49:43.594483 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:43.594444    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:45.397320 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.397292    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:49:45.464462 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.464428    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls\") pod \"b9d58bb1-400e-401d-b130-f384f4fddf79\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") "
Jul 02 13:49:45.464615 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.464494    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d58bb1-400e-401d-b130-f384f4fddf79-openshift-service-ca-bundle\") pod \"b9d58bb1-400e-401d-b130-f384f4fddf79\" (UID: \"b9d58bb1-400e-401d-b130-f384f4fddf79\") "
Jul 02 13:49:45.464862 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.464839    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d58bb1-400e-401d-b130-f384f4fddf79-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b9d58bb1-400e-401d-b130-f384f4fddf79" (UID: "b9d58bb1-400e-401d-b130-f384f4fddf79"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:49:45.466439 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.466420    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9d58bb1-400e-401d-b130-f384f4fddf79" (UID: "b9d58bb1-400e-401d-b130-f384f4fddf79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:49:45.565192 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.565109    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d58bb1-400e-401d-b130-f384f4fddf79-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:45.565192 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.565135    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d58bb1-400e-401d-b130-f384f4fddf79-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:45.874230 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.874196    2555 generic.go:356] "Generic (PLEG): container finished" podID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerID="6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a" exitCode=0
Jul 02 13:49:45.874397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.874242    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"
Jul 02 13:49:45.874397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.874281    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" event={"ID":"b9d58bb1-400e-401d-b130-f384f4fddf79","Type":"ContainerDied","Data":"6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a"}
Jul 02 13:49:45.874397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.874332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt" event={"ID":"b9d58bb1-400e-401d-b130-f384f4fddf79","Type":"ContainerDied","Data":"036ccc5cd8a104e2bb129a53d546a42d77a78dfefa1b23de3d5d7e4f2c06b80b"}
Jul 02 13:49:45.874397 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.874354    2555 scope.go:117] "RemoveContainer" containerID="6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a"
Jul 02 13:49:45.882568 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.882550    2555 scope.go:117] "RemoveContainer" containerID="6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a"
Jul 02 13:49:45.882843 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:49:45.882825    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a\": container with ID starting with 6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a not found: ID does not exist" containerID="6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a"
Jul 02 13:49:45.882921 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.882853    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a"} err="failed to get container status \"6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a\": rpc error: code = NotFound desc = could not find container \"6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a\": container with ID starting with 6f864c0787d1e502cc3ecc9d5b873df826efe94c7992c6121bf74259831eca8a not found: ID does not exist"
Jul 02 13:49:45.895677 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.895655    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"]
Jul 02 13:49:45.901531 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:45.901511    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-e2bd2-fc854767f-spdpt"]
Jul 02 13:49:47.225814 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:47.225777    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" path="/var/lib/kubelet/pods/b9d58bb1-400e-401d-b130-f384f4fddf79/volumes"
Jul 02 13:49:49.895465 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:49.895429    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"]
Jul 02 13:49:49.895951 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:49.895670    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c" containerID="cri-o://173a3d65eab0b8bf4245770dea298ca6ed31146e9a1fd5b2a25f7f961f6dfe5a" gracePeriod=30
Jul 02 13:49:50.036837 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.036808    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"]
Jul 02 13:49:50.037034 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.037011    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" containerID="cri-o://687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed" gracePeriod=30
Jul 02 13:49:50.037094 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.037064    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kube-rbac-proxy" containerID="cri-o://9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31" gracePeriod=30
Jul 02 13:49:50.172906 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.172872    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"]
Jul 02 13:49:50.173242 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173229    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kube-rbac-proxy"
Jul 02 13:49:50.173286 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173244    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kube-rbac-proxy"
Jul 02 13:49:50.173286 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173258    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2"
Jul 02 13:49:50.173286 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173264    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2"
Jul 02 13:49:50.173286 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173281    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container"
Jul 02 13:49:50.173414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173288    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container"
Jul 02 13:49:50.173414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173344    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kserve-container"
Jul 02 13:49:50.173414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173352    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e000058e-9278-448d-8993-fd666a913d9e" containerName="kube-rbac-proxy"
Jul 02 13:49:50.173414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.173358    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d58bb1-400e-401d-b130-f384f4fddf79" containerName="ensemble-graph-e2bd2"
Jul 02 13:49:50.176417 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.176401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.178822 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.178802    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7f184-predictor-serving-cert\""
Jul 02 13:49:50.179221 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.179203    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7f184-kube-rbac-proxy-sar-config\""
Jul 02 13:49:50.184449 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.184429    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"]
Jul 02 13:49:50.301520 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.301484    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.301673 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.301540    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.301673 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.301595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6977w\" (UniqueName: \"kubernetes.io/projected/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-api-access-6977w\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.402648 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.402613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.402821 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.402667    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.402821 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.402696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6977w\" (UniqueName: \"kubernetes.io/projected/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-api-access-6977w\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.402821 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:49:50.402784    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7f184-predictor-serving-cert: secret "error-404-isvc-7f184-predictor-serving-cert" not found
Jul 02 13:49:50.402998 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:49:50.402854    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls podName:09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7 nodeName:}" failed. No retries permitted until 2026-07-02 13:49:50.902834455 +0000 UTC m=+2098.093164120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls") pod "error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" (UID: "09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7") : secret "error-404-isvc-7f184-predictor-serving-cert" not found
Jul 02 13:49:50.403345 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.403326    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.411149 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.411094    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6977w\" (UniqueName: \"kubernetes.io/projected/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-api-access-6977w\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.900980 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.900948    2555 generic.go:356] "Generic (PLEG): container finished" podID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerID="9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31" exitCode=2
Jul 02 13:49:50.901326 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.901015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" event={"ID":"70a269e0-b1a9-40d8-b7a7-c4bdce124a68","Type":"ContainerDied","Data":"9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31"}
Jul 02 13:49:50.906262 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.906242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:50.908555 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:50.908526    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls\") pod \"error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:51.087598 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.087556    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:51.217604 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.217474    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"]
Jul 02 13:49:51.220385 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:49:51.220360    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ac4a1b_aff2_4e55_9f19_87b60d5ad4a7.slice/crio-2d7093590fb4a81980e1a653b1f0ca9015abe5484780f509eb9ca02a33a886ed WatchSource:0}: Error finding container 2d7093590fb4a81980e1a653b1f0ca9015abe5484780f509eb9ca02a33a886ed: Status 404 returned error can't find the container with id 2d7093590fb4a81980e1a653b1f0ca9015abe5484780f509eb9ca02a33a886ed
Jul 02 13:49:51.761555 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.761513    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:51.906956 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.906925    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" event={"ID":"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7","Type":"ContainerStarted","Data":"32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc"}
Jul 02 13:49:51.907437 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.906979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" event={"ID":"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7","Type":"ContainerStarted","Data":"ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f"}
Jul 02 13:49:51.907437 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.906999    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" event={"ID":"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7","Type":"ContainerStarted","Data":"2d7093590fb4a81980e1a653b1f0ca9015abe5484780f509eb9ca02a33a886ed"}
Jul 02 13:49:51.907437 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.907085    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:51.923645 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:51.923595    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podStartSLOduration=1.9235826089999999 podStartE2EDuration="1.923582609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:49:51.922799224 +0000 UTC m=+2099.113128897" watchObservedRunningTime="2026-07-02 13:49:51.923582609 +0000 UTC m=+2099.113912279"
Jul 02 13:49:52.755862 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:52.755820    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 02 13:49:52.916844 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:52.916811    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:52.918399 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:52.918375    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 02 13:49:53.320313 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.320278    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:49:53.323870 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.323849    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:49:53.324440 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.324425    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:49:53.328130 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.328111    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:49:53.501007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.500970    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 02 13:49:53.603323 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.603301    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:49:53.724457 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.724430    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljq74\" (UniqueName: \"kubernetes.io/projected/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-api-access-ljq74\") pod \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") "
Jul 02 13:49:53.724651 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.724477    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-proxy-tls\") pod \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") "
Jul 02 13:49:53.724651 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.724512    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-rbac-proxy-sar-config\") pod \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\" (UID: \"70a269e0-b1a9-40d8-b7a7-c4bdce124a68\") "
Jul 02 13:49:53.724951 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.724928    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70a269e0-b1a9-40d8-b7a7-c4bdce124a68" (UID: "70a269e0-b1a9-40d8-b7a7-c4bdce124a68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:49:53.726458 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.726438    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-api-access-ljq74" (OuterVolumeSpecName: "kube-api-access-ljq74") pod "70a269e0-b1a9-40d8-b7a7-c4bdce124a68" (UID: "70a269e0-b1a9-40d8-b7a7-c4bdce124a68"). InnerVolumeSpecName "kube-api-access-ljq74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:49:53.726528 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.726441    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70a269e0-b1a9-40d8-b7a7-c4bdce124a68" (UID: "70a269e0-b1a9-40d8-b7a7-c4bdce124a68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:49:53.826033 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.826000    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljq74\" (UniqueName: \"kubernetes.io/projected/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-api-access-ljq74\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:53.826033 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.826029    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:53.826033 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.826038    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70a269e0-b1a9-40d8-b7a7-c4bdce124a68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:49:53.921941 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.921857    2555 generic.go:356] "Generic (PLEG): container finished" podID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerID="687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed" exitCode=0
Jul 02 13:49:53.921941 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.921907    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"
Jul 02 13:49:53.922445 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.921936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" event={"ID":"70a269e0-b1a9-40d8-b7a7-c4bdce124a68","Type":"ContainerDied","Data":"687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed"}
Jul 02 13:49:53.922445 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.922010    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" event={"ID":"70a269e0-b1a9-40d8-b7a7-c4bdce124a68","Type":"ContainerDied","Data":"10a9c1f840400ca23ecd7f76e300bc160c1fb3514cbb633dc295494f48e2890e"}
Jul 02 13:49:53.922445 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.922051    2555 scope.go:117] "RemoveContainer" containerID="9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31"
Jul 02 13:49:53.922649 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.922626    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 02 13:49:53.930904 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.930888    2555 scope.go:117] "RemoveContainer" containerID="687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed"
Jul 02 13:49:53.937890 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.937815    2555 scope.go:117] "RemoveContainer" containerID="9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31"
Jul 02 13:49:53.938066 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:49:53.938049    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31\": container with ID starting with 9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31 not found: ID does not exist" containerID="9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31"
Jul 02 13:49:53.938122 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.938073    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31"} err="failed to get container status \"9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31\": rpc error: code = NotFound desc = could not find container \"9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31\": container with ID starting with 9f475f05c463ffbebd96a535f0ef30fbc03a0991670004ebed9a55e5f0665f31 not found: ID does not exist"
Jul 02 13:49:53.938122 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.938089    2555 scope.go:117] "RemoveContainer" containerID="687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed"
Jul 02 13:49:53.938310 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:49:53.938296    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed\": container with ID starting with 687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed not found: ID does not exist" containerID="687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed"
Jul 02 13:49:53.938355 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.938313    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed"} err="failed to get container status \"687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed\": rpc error: code = NotFound desc = could not find container \"687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed\": container with ID starting with 687b90ae6547fdd46b72466e1dc956c70aa292a6c63264b2775fb828a9128fed not found: ID does not exist"
Jul 02 13:49:53.942977 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.942956    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"]
Jul 02 13:49:53.948007 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:53.947987    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd"]
Jul 02 13:49:54.507741 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:54.507682    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7b15c-predictor-6c577469b4-7vqmd" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: i/o timeout"
Jul 02 13:49:55.225368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:55.225336    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" path="/var/lib/kubelet/pods/70a269e0-b1a9-40d8-b7a7-c4bdce124a68/volumes"
Jul 02 13:49:56.761860 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:56.761826    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:49:58.927258 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:58.927225    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:49:58.927909 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:49:58.927880    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 02 13:50:01.761815 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:01.761780    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:01.762221 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:01.761887    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:50:02.756302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:02.756276    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:50:06.762075 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:06.762030    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:08.928634 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:08.928596    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 02 13:50:11.762188 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:11.762148    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:15.467106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.467074    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"]
Jul 02 13:50:15.467511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.467383    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container"
Jul 02 13:50:15.467511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.467394    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container"
Jul 02 13:50:15.467511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.467422    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kube-rbac-proxy"
Jul 02 13:50:15.467511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.467429    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kube-rbac-proxy"
Jul 02 13:50:15.467511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.467492    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kube-rbac-proxy"
Jul 02 13:50:15.467511 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.467501    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a269e0-b1a9-40d8-b7a7-c4bdce124a68" containerName="kserve-container"
Jul 02 13:50:15.470306 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.470286    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:15.472852 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.472830    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9633b-serving-cert\""
Jul 02 13:50:15.472979 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.472961    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-9633b-kube-rbac-proxy-sar-config\""
Jul 02 13:50:15.478522 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.478502    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"]
Jul 02 13:50:15.500972 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.500940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2ad189-e2e8-400c-be38-cae7f926895d-openshift-service-ca-bundle\") pod \"splitter-graph-9633b-986484644-tkj85\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") " pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:15.501094 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.501001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls\") pod \"splitter-graph-9633b-986484644-tkj85\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") " pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:15.602150 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.602120    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2ad189-e2e8-400c-be38-cae7f926895d-openshift-service-ca-bundle\") pod \"splitter-graph-9633b-986484644-tkj85\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") " pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:15.602339 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.602169    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls\") pod \"splitter-graph-9633b-986484644-tkj85\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") " pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:15.602339 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:15.602280    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-9633b-serving-cert: secret "splitter-graph-9633b-serving-cert" not found
Jul 02 13:50:15.602339 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:15.602340    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls podName:3d2ad189-e2e8-400c-be38-cae7f926895d nodeName:}" failed. No retries permitted until 2026-07-02 13:50:16.102324488 +0000 UTC m=+2123.292654137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls") pod "splitter-graph-9633b-986484644-tkj85" (UID: "3d2ad189-e2e8-400c-be38-cae7f926895d") : secret "splitter-graph-9633b-serving-cert" not found
Jul 02 13:50:15.603202 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:15.603175    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2ad189-e2e8-400c-be38-cae7f926895d-openshift-service-ca-bundle\") pod \"splitter-graph-9633b-986484644-tkj85\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") " pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:16.106460 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:16.106425    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls\") pod \"splitter-graph-9633b-986484644-tkj85\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") " pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:16.108756 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:16.108730    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls\") pod \"splitter-graph-9633b-986484644-tkj85\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") " pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:16.380831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:16.380743    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:16.508320 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:16.508293    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"]
Jul 02 13:50:16.510478 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:50:16.510452    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2ad189_e2e8_400c_be38_cae7f926895d.slice/crio-38b27ff61b17038e6373c29662bc3eb9f3f1664a45a95235281bbf75cabe93f8 WatchSource:0}: Error finding container 38b27ff61b17038e6373c29662bc3eb9f3f1664a45a95235281bbf75cabe93f8: Status 404 returned error can't find the container with id 38b27ff61b17038e6373c29662bc3eb9f3f1664a45a95235281bbf75cabe93f8
Jul 02 13:50:16.761461 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:16.761366    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:17.027018 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:17.026918    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" event={"ID":"3d2ad189-e2e8-400c-be38-cae7f926895d","Type":"ContainerStarted","Data":"4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7"}
Jul 02 13:50:17.027018 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:17.026995    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" event={"ID":"3d2ad189-e2e8-400c-be38-cae7f926895d","Type":"ContainerStarted","Data":"38b27ff61b17038e6373c29662bc3eb9f3f1664a45a95235281bbf75cabe93f8"}
Jul 02 13:50:17.027261 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:17.027025    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:17.045586 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:17.045534    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podStartSLOduration=2.045519935 podStartE2EDuration="2.045519935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:50:17.043149937 +0000 UTC m=+2124.233479608" watchObservedRunningTime="2026-07-02 13:50:17.045519935 +0000 UTC m=+2124.235849606"
Jul 02 13:50:18.928119 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:18.928075    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 02 13:50:20.041513 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.041483    2555 generic.go:356] "Generic (PLEG): container finished" podID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerID="173a3d65eab0b8bf4245770dea298ca6ed31146e9a1fd5b2a25f7f961f6dfe5a" exitCode=137
Jul 02 13:50:20.041893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.041552    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" event={"ID":"f3faf2de-97cd-44c2-9534-2c6d935c0447","Type":"ContainerDied","Data":"173a3d65eab0b8bf4245770dea298ca6ed31146e9a1fd5b2a25f7f961f6dfe5a"}
Jul 02 13:50:20.041893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.041600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p" event={"ID":"f3faf2de-97cd-44c2-9534-2c6d935c0447","Type":"ContainerDied","Data":"f3b5d81b1714f929b89d875918803e86dd2430c04d22cf0afe39328fad804630"}
Jul 02 13:50:20.041893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.041617    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b5d81b1714f929b89d875918803e86dd2430c04d22cf0afe39328fad804630"
Jul 02 13:50:20.042065 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.042006    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:50:20.133443 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.133417    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3faf2de-97cd-44c2-9534-2c6d935c0447-proxy-tls\") pod \"f3faf2de-97cd-44c2-9534-2c6d935c0447\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") "
Jul 02 13:50:20.133593 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.133470    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3faf2de-97cd-44c2-9534-2c6d935c0447-openshift-service-ca-bundle\") pod \"f3faf2de-97cd-44c2-9534-2c6d935c0447\" (UID: \"f3faf2de-97cd-44c2-9534-2c6d935c0447\") "
Jul 02 13:50:20.133862 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.133831    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3faf2de-97cd-44c2-9534-2c6d935c0447-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f3faf2de-97cd-44c2-9534-2c6d935c0447" (UID: "f3faf2de-97cd-44c2-9534-2c6d935c0447"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:50:20.135386 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.135363    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3faf2de-97cd-44c2-9534-2c6d935c0447-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3faf2de-97cd-44c2-9534-2c6d935c0447" (UID: "f3faf2de-97cd-44c2-9534-2c6d935c0447"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:50:20.234115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.234036    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3faf2de-97cd-44c2-9534-2c6d935c0447-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:50:20.234115 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:20.234066    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3faf2de-97cd-44c2-9534-2c6d935c0447-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:50:21.045450 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:21.045417    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"
Jul 02 13:50:21.064624 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:21.064598    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"]
Jul 02 13:50:21.073148 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:21.068373    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-7b15c-657959bb75-qgx8p"]
Jul 02 13:50:21.226229 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:21.226197    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" path="/var/lib/kubelet/pods/f3faf2de-97cd-44c2-9534-2c6d935c0447/volumes"
Jul 02 13:50:23.037211 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:23.037183    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:25.550089 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.549998    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"]
Jul 02 13:50:25.550564 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.550223    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b" containerID="cri-o://4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7" gracePeriod=30
Jul 02 13:50:25.748861 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.748830    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"]
Jul 02 13:50:25.749227 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.749209    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c"
Jul 02 13:50:25.749320 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.749229    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c"
Jul 02 13:50:25.749379 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.749342    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3faf2de-97cd-44c2-9534-2c6d935c0447" containerName="sequence-graph-7b15c"
Jul 02 13:50:25.753656 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.753635    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.756046 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.756003    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-03d0f-predictor-serving-cert\""
Jul 02 13:50:25.756158 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.756022    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-03d0f-kube-rbac-proxy-sar-config\""
Jul 02 13:50:25.762032 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.762010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"]
Jul 02 13:50:25.778472 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.778448    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rb28\" (UniqueName: \"kubernetes.io/projected/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-api-access-8rb28\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.778606 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.778501    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.778606 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.778577    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.801828 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.801754    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"]
Jul 02 13:50:25.802004 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.801977    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container" containerID="cri-o://5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b" gracePeriod=30
Jul 02 13:50:25.802080 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.802001    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kube-rbac-proxy" containerID="cri-o://d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378" gracePeriod=30
Jul 02 13:50:25.879989 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.879952    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rb28\" (UniqueName: \"kubernetes.io/projected/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-api-access-8rb28\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.880218 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.880037    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.880218 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.880089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.880345 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:25.880220    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-serving-cert: secret "error-404-isvc-03d0f-predictor-serving-cert" not found
Jul 02 13:50:25.880345 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:25.880289    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls podName:1b43aeb7-3760-4843-b59b-718ce64e0011 nodeName:}" failed. No retries permitted until 2026-07-02 13:50:26.380269639 +0000 UTC m=+2133.570599308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls") pod "error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" (UID: "1b43aeb7-3760-4843-b59b-718ce64e0011") : secret "error-404-isvc-03d0f-predictor-serving-cert" not found
Jul 02 13:50:25.880699 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.880677    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:25.888820 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:25.888794    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rb28\" (UniqueName: \"kubernetes.io/projected/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-api-access-8rb28\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:26.070189 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:26.070096    2555 generic.go:356] "Generic (PLEG): container finished" podID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerID="d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378" exitCode=2
Jul 02 13:50:26.070189 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:26.070137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" event={"ID":"2df728f4-3c51-46d9-bcf9-40f3da845c0a","Type":"ContainerDied","Data":"d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378"}
Jul 02 13:50:26.384114 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:26.384063    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:26.386486 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:26.386459    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls\") pod \"error-404-isvc-03d0f-predictor-59b6876b74-9wdb4\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") " pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:26.664698 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:26.664611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:26.794333 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:26.794309    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"]
Jul 02 13:50:26.796797 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:50:26.796748    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b43aeb7_3760_4843_b59b_718ce64e0011.slice/crio-ee023417ba241b3ed3fc77875794209b5d9727efb672cfa70d3ee35ea6da4010 WatchSource:0}: Error finding container ee023417ba241b3ed3fc77875794209b5d9727efb672cfa70d3ee35ea6da4010: Status 404 returned error can't find the container with id ee023417ba241b3ed3fc77875794209b5d9727efb672cfa70d3ee35ea6da4010
Jul 02 13:50:27.077274 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.077241    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" event={"ID":"1b43aeb7-3760-4843-b59b-718ce64e0011","Type":"ContainerStarted","Data":"7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc"}
Jul 02 13:50:27.077466 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.077386    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:27.077466 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.077436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" event={"ID":"1b43aeb7-3760-4843-b59b-718ce64e0011","Type":"ContainerStarted","Data":"e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb"}
Jul 02 13:50:27.077556 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.077479    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:27.077556 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.077497    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" event={"ID":"1b43aeb7-3760-4843-b59b-718ce64e0011","Type":"ContainerStarted","Data":"ee023417ba241b3ed3fc77875794209b5d9727efb672cfa70d3ee35ea6da4010"}
Jul 02 13:50:27.078988 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.078962    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 13:50:27.095168 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.095123    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podStartSLOduration=2.095108159 podStartE2EDuration="2.095108159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:50:27.093228479 +0000 UTC m=+2134.283558149" watchObservedRunningTime="2026-07-02 13:50:27.095108159 +0000 UTC m=+2134.285437827"
Jul 02 13:50:27.751247 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:27.751207    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 02 13:50:28.035368 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.035271    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:28.082321 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.082286    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 13:50:28.857685 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.857663    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:50:28.904886 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.904849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4cbt\" (UniqueName: \"kubernetes.io/projected/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-api-access-s4cbt\") pod \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") "
Jul 02 13:50:28.904886 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.904891    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df728f4-3c51-46d9-bcf9-40f3da845c0a-proxy-tls\") pod \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") "
Jul 02 13:50:28.905139 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.904912    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-rbac-proxy-sar-config\") pod \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\" (UID: \"2df728f4-3c51-46d9-bcf9-40f3da845c0a\") "
Jul 02 13:50:28.905321 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.905299    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2df728f4-3c51-46d9-bcf9-40f3da845c0a" (UID: "2df728f4-3c51-46d9-bcf9-40f3da845c0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:50:28.906967 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.906940    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df728f4-3c51-46d9-bcf9-40f3da845c0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2df728f4-3c51-46d9-bcf9-40f3da845c0a" (UID: "2df728f4-3c51-46d9-bcf9-40f3da845c0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:50:28.907072 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.906974    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-api-access-s4cbt" (OuterVolumeSpecName: "kube-api-access-s4cbt") pod "2df728f4-3c51-46d9-bcf9-40f3da845c0a" (UID: "2df728f4-3c51-46d9-bcf9-40f3da845c0a"). InnerVolumeSpecName "kube-api-access-s4cbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:50:28.928739 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:28.928686    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 02 13:50:29.006291 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.006208    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4cbt\" (UniqueName: \"kubernetes.io/projected/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-api-access-s4cbt\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:50:29.006291 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.006235    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2df728f4-3c51-46d9-bcf9-40f3da845c0a-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:50:29.006291 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.006244    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2df728f4-3c51-46d9-bcf9-40f3da845c0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:50:29.088374 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.088337    2555 generic.go:356] "Generic (PLEG): container finished" podID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerID="5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b" exitCode=0
Jul 02 13:50:29.088534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.088389    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"
Jul 02 13:50:29.088534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.088407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" event={"ID":"2df728f4-3c51-46d9-bcf9-40f3da845c0a","Type":"ContainerDied","Data":"5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b"}
Jul 02 13:50:29.088534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.088466    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5" event={"ID":"2df728f4-3c51-46d9-bcf9-40f3da845c0a","Type":"ContainerDied","Data":"3a0f4687d36e2184a21707d166c4289c7ce0b0f1c771f0a242b422188e88c622"}
Jul 02 13:50:29.088534 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.088493    2555 scope.go:117] "RemoveContainer" containerID="d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378"
Jul 02 13:50:29.097485 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.097466    2555 scope.go:117] "RemoveContainer" containerID="5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b"
Jul 02 13:50:29.104811 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.104794    2555 scope.go:117] "RemoveContainer" containerID="d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378"
Jul 02 13:50:29.105039 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:29.105024    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378\": container with ID starting with d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378 not found: ID does not exist" containerID="d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378"
Jul 02 13:50:29.105082 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.105046    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378"} err="failed to get container status \"d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378\": rpc error: code = NotFound desc = could not find container \"d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378\": container with ID starting with d583ec20a94bb2f9b24f5057fe8607392ab98f9c056b6dbf4d4c9fc5626ab378 not found: ID does not exist"
Jul 02 13:50:29.105082 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.105062    2555 scope.go:117] "RemoveContainer" containerID="5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b"
Jul 02 13:50:29.105297 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:29.105280    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b\": container with ID starting with 5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b not found: ID does not exist" containerID="5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b"
Jul 02 13:50:29.105335 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.105304    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b"} err="failed to get container status \"5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b\": rpc error: code = NotFound desc = could not find container \"5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b\": container with ID starting with 5cbef13945bd6269755882d99654b8917c69ebd2e073bd9d0897dc48522b486b not found: ID does not exist"
Jul 02 13:50:29.110039 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.110018    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"]
Jul 02 13:50:29.113870 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.113846    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9633b-predictor-6db75d69f8-hx5s5"]
Jul 02 13:50:29.225632 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:29.225603    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" path="/var/lib/kubelet/pods/2df728f4-3c51-46d9-bcf9-40f3da845c0a/volumes"
Jul 02 13:50:33.035661 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:33.035621    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:33.088138 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:33.088113    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:50:33.088689 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:33.088653    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 13:50:38.035133 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:38.035095    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:38.035528 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:38.035209    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:38.928855 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:38.928828    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 13:50:43.035474 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:43.035434    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:43.088837 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:43.088806    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 13:50:48.035690 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:48.035609    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:50.108356 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.108316    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"]
Jul 02 13:50:50.108864 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.108632    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kube-rbac-proxy"
Jul 02 13:50:50.108864 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.108643    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kube-rbac-proxy"
Jul 02 13:50:50.108864 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.108663    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container"
Jul 02 13:50:50.108864 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.108669    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container"
Jul 02 13:50:50.108864 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.108742    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kserve-container"
Jul 02 13:50:50.108864 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.108751    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df728f4-3c51-46d9-bcf9-40f3da845c0a" containerName="kube-rbac-proxy"
Jul 02 13:50:50.112732 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.112696    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:50.115118 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.115096    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7f184-serving-cert\""
Jul 02 13:50:50.115307 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.115144    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-7f184-kube-rbac-proxy-sar-config\""
Jul 02 13:50:50.121156 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.121130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"]
Jul 02 13:50:50.183113 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.183072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/175b54bb-8dce-4f40-9044-70aef274b1ae-openshift-service-ca-bundle\") pod \"switch-graph-7f184-7676f88dd9-z4frx\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") " pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:50.183310 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.183129    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls\") pod \"switch-graph-7f184-7676f88dd9-z4frx\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") " pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:50.284531 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.284499    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/175b54bb-8dce-4f40-9044-70aef274b1ae-openshift-service-ca-bundle\") pod \"switch-graph-7f184-7676f88dd9-z4frx\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") " pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:50.284741 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.284548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls\") pod \"switch-graph-7f184-7676f88dd9-z4frx\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") " pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:50.284826 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:50.284744    2555 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-7f184-serving-cert: secret "switch-graph-7f184-serving-cert" not found
Jul 02 13:50:50.284892 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:50.284827    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls podName:175b54bb-8dce-4f40-9044-70aef274b1ae nodeName:}" failed. No retries permitted until 2026-07-02 13:50:50.784805504 +0000 UTC m=+2157.975135165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls") pod "switch-graph-7f184-7676f88dd9-z4frx" (UID: "175b54bb-8dce-4f40-9044-70aef274b1ae") : secret "switch-graph-7f184-serving-cert" not found
Jul 02 13:50:50.285199 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.285176    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/175b54bb-8dce-4f40-9044-70aef274b1ae-openshift-service-ca-bundle\") pod \"switch-graph-7f184-7676f88dd9-z4frx\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") " pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:50.788146 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.788107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls\") pod \"switch-graph-7f184-7676f88dd9-z4frx\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") " pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:50.790400 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:50.790371    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls\") pod \"switch-graph-7f184-7676f88dd9-z4frx\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") " pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:51.023975 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:51.023950    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:51.153193 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:51.153163    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"]
Jul 02 13:50:51.156576 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:50:51.156549    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175b54bb_8dce_4f40_9044_70aef274b1ae.slice/crio-368cac13b5e262d97f17dac360211c76ec107aeef266f3089f3a87c8278cc8ef WatchSource:0}: Error finding container 368cac13b5e262d97f17dac360211c76ec107aeef266f3089f3a87c8278cc8ef: Status 404 returned error can't find the container with id 368cac13b5e262d97f17dac360211c76ec107aeef266f3089f3a87c8278cc8ef
Jul 02 13:50:51.195438 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:51.195412    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" event={"ID":"175b54bb-8dce-4f40-9044-70aef274b1ae","Type":"ContainerStarted","Data":"368cac13b5e262d97f17dac360211c76ec107aeef266f3089f3a87c8278cc8ef"}
Jul 02 13:50:52.200790 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:52.200749    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" event={"ID":"175b54bb-8dce-4f40-9044-70aef274b1ae","Type":"ContainerStarted","Data":"1bfe8c95096d50dbb8f0f9feb31ea571f4e0ff59f5fe386083579f63ba5c1cbd"}
Jul 02 13:50:52.201265 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:52.200816    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:50:52.216970 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:52.216923    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podStartSLOduration=2.216910466 podStartE2EDuration="2.216910466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:50:52.215278527 +0000 UTC m=+2159.405608199" watchObservedRunningTime="2026-07-02 13:50:52.216910466 +0000 UTC m=+2159.407240138"
Jul 02 13:50:53.035169 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:53.035135    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:50:53.089221 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:53.089180    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 13:50:53.450806 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:53.450773    2555 scope.go:117] "RemoveContainer" containerID="173a3d65eab0b8bf4245770dea298ca6ed31146e9a1fd5b2a25f7f961f6dfe5a"
Jul 02 13:50:56.205439 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.205414    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:56.218057 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.218031    2555 generic.go:356] "Generic (PLEG): container finished" podID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerID="4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7" exitCode=0
Jul 02 13:50:56.218161 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.218066    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"
Jul 02 13:50:56.218161 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.218110    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" event={"ID":"3d2ad189-e2e8-400c-be38-cae7f926895d","Type":"ContainerDied","Data":"4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7"}
Jul 02 13:50:56.218161 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.218155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85" event={"ID":"3d2ad189-e2e8-400c-be38-cae7f926895d","Type":"ContainerDied","Data":"38b27ff61b17038e6373c29662bc3eb9f3f1664a45a95235281bbf75cabe93f8"}
Jul 02 13:50:56.218263 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.218176    2555 scope.go:117] "RemoveContainer" containerID="4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7"
Jul 02 13:50:56.228656 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.228614    2555 scope.go:117] "RemoveContainer" containerID="4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7"
Jul 02 13:50:56.229168 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:50:56.229143    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7\": container with ID starting with 4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7 not found: ID does not exist" containerID="4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7"
Jul 02 13:50:56.229269 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.229180    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7"} err="failed to get container status \"4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7\": rpc error: code = NotFound desc = could not find container \"4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7\": container with ID starting with 4e08c5e2bb389781c1beced02f92cbd791c8fc896997861c6e5962bf8d7d6ca7 not found: ID does not exist"
Jul 02 13:50:56.334237 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.334215    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2ad189-e2e8-400c-be38-cae7f926895d-openshift-service-ca-bundle\") pod \"3d2ad189-e2e8-400c-be38-cae7f926895d\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") "
Jul 02 13:50:56.334338 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.334263    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls\") pod \"3d2ad189-e2e8-400c-be38-cae7f926895d\" (UID: \"3d2ad189-e2e8-400c-be38-cae7f926895d\") "
Jul 02 13:50:56.334533 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.334511    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2ad189-e2e8-400c-be38-cae7f926895d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3d2ad189-e2e8-400c-be38-cae7f926895d" (UID: "3d2ad189-e2e8-400c-be38-cae7f926895d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:50:56.336163 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.336144    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d2ad189-e2e8-400c-be38-cae7f926895d" (UID: "3d2ad189-e2e8-400c-be38-cae7f926895d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:50:56.434939 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.434915    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d2ad189-e2e8-400c-be38-cae7f926895d-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:50:56.434939 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.434935    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d2ad189-e2e8-400c-be38-cae7f926895d-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:50:56.539414 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.539386    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"]
Jul 02 13:50:56.543655 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:56.543633    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-9633b-986484644-tkj85"]
Jul 02 13:50:57.228373 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:57.228332    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" path="/var/lib/kubelet/pods/3d2ad189-e2e8-400c-be38-cae7f926895d/volumes"
Jul 02 13:50:58.211952 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:50:58.211923    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 13:51:03.088987 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:03.088948    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 13:51:13.089375 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:13.089349    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:51:25.780350 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.780318    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"]
Jul 02 13:51:25.780787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.780643    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b"
Jul 02 13:51:25.780787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.780654    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b"
Jul 02 13:51:25.780787 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.780731    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2ad189-e2e8-400c-be38-cae7f926895d" containerName="splitter-graph-9633b"
Jul 02 13:51:25.783637 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.783621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:25.787012 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.786990    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-03d0f-serving-cert\""
Jul 02 13:51:25.787362 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.787340    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-03d0f-kube-rbac-proxy-sar-config\""
Jul 02 13:51:25.798353 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.798331    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"]
Jul 02 13:51:25.864045 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.864011    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5462ee6-1101-4006-ac12-c3069ae275b7-openshift-service-ca-bundle\") pod \"splitter-graph-03d0f-7b76878d5c-rzw2c\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") " pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:25.864220 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.864064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5462ee6-1101-4006-ac12-c3069ae275b7-proxy-tls\") pod \"splitter-graph-03d0f-7b76878d5c-rzw2c\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") " pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:25.964822 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.964787    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5462ee6-1101-4006-ac12-c3069ae275b7-openshift-service-ca-bundle\") pod \"splitter-graph-03d0f-7b76878d5c-rzw2c\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") " pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:25.965003 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.964840    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5462ee6-1101-4006-ac12-c3069ae275b7-proxy-tls\") pod \"splitter-graph-03d0f-7b76878d5c-rzw2c\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") " pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:25.965407 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.965383    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5462ee6-1101-4006-ac12-c3069ae275b7-openshift-service-ca-bundle\") pod \"splitter-graph-03d0f-7b76878d5c-rzw2c\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") " pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:25.967201 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:25.967182    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5462ee6-1101-4006-ac12-c3069ae275b7-proxy-tls\") pod \"splitter-graph-03d0f-7b76878d5c-rzw2c\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") " pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:26.094146 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:26.094119    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:26.218485 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:26.218458    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"]
Jul 02 13:51:26.221053 ip-10-0-133-122 kubenswrapper[2555]: W0702 13:51:26.221021    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5462ee6_1101_4006_ac12_c3069ae275b7.slice/crio-c1a22e8678ff1382cb71f5b70caabd78a74ccf6dd684c20608f3a453810a801b WatchSource:0}: Error finding container c1a22e8678ff1382cb71f5b70caabd78a74ccf6dd684c20608f3a453810a801b: Status 404 returned error can't find the container with id c1a22e8678ff1382cb71f5b70caabd78a74ccf6dd684c20608f3a453810a801b
Jul 02 13:51:26.344831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:26.344755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" event={"ID":"d5462ee6-1101-4006-ac12-c3069ae275b7","Type":"ContainerStarted","Data":"dafc05c9ef7de27bac630b2418165bf622fd8b7b2a26db72a6c941853fa5931a"}
Jul 02 13:51:26.344831 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:26.344807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" event={"ID":"d5462ee6-1101-4006-ac12-c3069ae275b7","Type":"ContainerStarted","Data":"c1a22e8678ff1382cb71f5b70caabd78a74ccf6dd684c20608f3a453810a801b"}
Jul 02 13:51:26.344982 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:26.344839    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:51:26.359339 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:26.359295    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podStartSLOduration=1.359282081 podStartE2EDuration="1.359282081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 13:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 13:51:26.357930745 +0000 UTC m=+2193.548260416" watchObservedRunningTime="2026-07-02 13:51:26.359282081 +0000 UTC m=+2193.549611751"
Jul 02 13:51:32.354817 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:51:32.354787    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:54:53.345670 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:54:53.345639    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:54:53.349734 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:54:53.349695    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:54:53.349927 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:54:53.349908    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:54:53.354253 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:54:53.354235    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:59:40.371832 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:40.371794    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"]
Jul 02 13:59:40.372418 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:40.372062    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f" containerID="cri-o://dafc05c9ef7de27bac630b2418165bf622fd8b7b2a26db72a6c941853fa5931a" gracePeriod=30
Jul 02 13:59:40.520750 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:40.520717    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"]
Jul 02 13:59:40.521041 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:40.520992    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" containerID="cri-o://e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb" gracePeriod=30
Jul 02 13:59:40.521162 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:40.521042    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kube-rbac-proxy" containerID="cri-o://7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc" gracePeriod=30
Jul 02 13:59:41.472893 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:41.472860    2555 generic.go:356] "Generic (PLEG): container finished" podID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerID="7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc" exitCode=2
Jul 02 13:59:41.473263 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:41.472931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" event={"ID":"1b43aeb7-3760-4843-b59b-718ce64e0011","Type":"ContainerDied","Data":"7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc"}
Jul 02 13:59:42.353894 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:42.353852    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:59:43.083098 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.083053    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 02 13:59:43.088924 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.088885    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 02 13:59:43.368421 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.368399    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:59:43.371531 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.371502    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rb28\" (UniqueName: \"kubernetes.io/projected/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-api-access-8rb28\") pod \"1b43aeb7-3760-4843-b59b-718ce64e0011\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") "
Jul 02 13:59:43.371612 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.371547    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-rbac-proxy-sar-config\") pod \"1b43aeb7-3760-4843-b59b-718ce64e0011\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") "
Jul 02 13:59:43.371612 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.371600    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls\") pod \"1b43aeb7-3760-4843-b59b-718ce64e0011\" (UID: \"1b43aeb7-3760-4843-b59b-718ce64e0011\") "
Jul 02 13:59:43.371990 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.371965    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b43aeb7-3760-4843-b59b-718ce64e0011" (UID: "1b43aeb7-3760-4843-b59b-718ce64e0011"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 13:59:43.373588 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.373564    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b43aeb7-3760-4843-b59b-718ce64e0011" (UID: "1b43aeb7-3760-4843-b59b-718ce64e0011"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 13:59:43.373588 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.373575    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-api-access-8rb28" (OuterVolumeSpecName: "kube-api-access-8rb28") pod "1b43aeb7-3760-4843-b59b-718ce64e0011" (UID: "1b43aeb7-3760-4843-b59b-718ce64e0011"). InnerVolumeSpecName "kube-api-access-8rb28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 13:59:43.472214 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.472184    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b43aeb7-3760-4843-b59b-718ce64e0011-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:59:43.472214 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.472208    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rb28\" (UniqueName: \"kubernetes.io/projected/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-api-access-8rb28\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:59:43.472214 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.472221    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b43aeb7-3760-4843-b59b-718ce64e0011-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 13:59:43.482106 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.482078    2555 generic.go:356] "Generic (PLEG): container finished" podID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerID="e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb" exitCode=0
Jul 02 13:59:43.482213 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.482126    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"
Jul 02 13:59:43.482213 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.482128    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" event={"ID":"1b43aeb7-3760-4843-b59b-718ce64e0011","Type":"ContainerDied","Data":"e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb"}
Jul 02 13:59:43.482213 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.482183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4" event={"ID":"1b43aeb7-3760-4843-b59b-718ce64e0011","Type":"ContainerDied","Data":"ee023417ba241b3ed3fc77875794209b5d9727efb672cfa70d3ee35ea6da4010"}
Jul 02 13:59:43.482213 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.482208    2555 scope.go:117] "RemoveContainer" containerID="7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc"
Jul 02 13:59:43.490594 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.490579    2555 scope.go:117] "RemoveContainer" containerID="e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb"
Jul 02 13:59:43.497488 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.497468    2555 scope.go:117] "RemoveContainer" containerID="7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc"
Jul 02 13:59:43.497773 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:59:43.497736    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc\": container with ID starting with 7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc not found: ID does not exist" containerID="7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc"
Jul 02 13:59:43.497867 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.497779    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc"} err="failed to get container status \"7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc\": rpc error: code = NotFound desc = could not find container \"7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc\": container with ID starting with 7d81d4f97513158cfc6fe8774d6f8657edcab36d1faa93147f41cad9a837e5bc not found: ID does not exist"
Jul 02 13:59:43.497867 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.497795    2555 scope.go:117] "RemoveContainer" containerID="e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb"
Jul 02 13:59:43.498050 ip-10-0-133-122 kubenswrapper[2555]: E0702 13:59:43.498033    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb\": container with ID starting with e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb not found: ID does not exist" containerID="e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb"
Jul 02 13:59:43.498091 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.498056    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb"} err="failed to get container status \"e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb\": rpc error: code = NotFound desc = could not find container \"e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb\": container with ID starting with e9f822b97eab03588804397b8ec3b3fbc4aac26d9c4ef00484c46634dd56fabb not found: ID does not exist"
Jul 02 13:59:43.502504 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.502485    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"]
Jul 02 13:59:43.508116 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:43.508097    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-03d0f-predictor-59b6876b74-9wdb4"]
Jul 02 13:59:45.225596 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:45.225504    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" path="/var/lib/kubelet/pods/1b43aeb7-3760-4843-b59b-718ce64e0011/volumes"
Jul 02 13:59:47.353041 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:47.353003    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:59:52.353811 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:52.353771    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 13:59:52.354206 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:52.353881    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 13:59:53.373529 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:53.373415    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:59:53.390538 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:53.377738    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 13:59:53.390538 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:53.377976    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:59:53.390538 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:53.382124    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 13:59:57.353302 ip-10-0-133-122 kubenswrapper[2555]: I0702 13:59:57.353264    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:02.353720 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:02.353668    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:07.352847 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:07.352805    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:00:10.597970 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:10.597938    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerID="dafc05c9ef7de27bac630b2418165bf622fd8b7b2a26db72a6c941853fa5931a" exitCode=0
Jul 02 14:00:10.598319 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:10.598010    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" event={"ID":"d5462ee6-1101-4006-ac12-c3069ae275b7","Type":"ContainerDied","Data":"dafc05c9ef7de27bac630b2418165bf622fd8b7b2a26db72a6c941853fa5931a"}
Jul 02 14:00:11.019247 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.019226    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 14:00:11.075501 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.075475    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5462ee6-1101-4006-ac12-c3069ae275b7-proxy-tls\") pod \"d5462ee6-1101-4006-ac12-c3069ae275b7\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") "
Jul 02 14:00:11.075624 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.075559    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5462ee6-1101-4006-ac12-c3069ae275b7-openshift-service-ca-bundle\") pod \"d5462ee6-1101-4006-ac12-c3069ae275b7\" (UID: \"d5462ee6-1101-4006-ac12-c3069ae275b7\") "
Jul 02 14:00:11.075897 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.075872    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5462ee6-1101-4006-ac12-c3069ae275b7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d5462ee6-1101-4006-ac12-c3069ae275b7" (UID: "d5462ee6-1101-4006-ac12-c3069ae275b7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:00:11.077484 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.077464    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5462ee6-1101-4006-ac12-c3069ae275b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5462ee6-1101-4006-ac12-c3069ae275b7" (UID: "d5462ee6-1101-4006-ac12-c3069ae275b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:00:11.176739 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.176639    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5462ee6-1101-4006-ac12-c3069ae275b7-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 14:00:11.176739 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.176665    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5462ee6-1101-4006-ac12-c3069ae275b7-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 14:00:11.602976 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.602939    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c" event={"ID":"d5462ee6-1101-4006-ac12-c3069ae275b7","Type":"ContainerDied","Data":"c1a22e8678ff1382cb71f5b70caabd78a74ccf6dd684c20608f3a453810a801b"}
Jul 02 14:00:11.602976 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.602958    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"
Jul 02 14:00:11.603440 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.603011    2555 scope.go:117] "RemoveContainer" containerID="dafc05c9ef7de27bac630b2418165bf622fd8b7b2a26db72a6c941853fa5931a"
Jul 02 14:00:11.618841 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.618818    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"]
Jul 02 14:00:11.621636 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:11.621615    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-03d0f-7b76878d5c-rzw2c"]
Jul 02 14:00:13.225641 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:00:13.225610    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" path="/var/lib/kubelet/pods/d5462ee6-1101-4006-ac12-c3069ae275b7/volumes"
Jul 02 14:04:53.400307 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:04:53.400195    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 14:04:53.404936 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:04:53.404918    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 14:04:53.405119 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:04:53.405099    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 14:04:53.409563 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:04:53.409546    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 14:07:09.446194 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:09.446159    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"]
Jul 02 14:07:09.446778 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:09.446385    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184" containerID="cri-o://1bfe8c95096d50dbb8f0f9feb31ea571f4e0ff59f5fe386083579f63ba5c1cbd" gracePeriod=30
Jul 02 14:07:09.638678 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:09.638635    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"]
Jul 02 14:07:09.638993 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:09.638935    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container" containerID="cri-o://ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f" gracePeriod=30
Jul 02 14:07:09.639129 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:09.638971    2555 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kube-rbac-proxy" containerID="cri-o://32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc" gracePeriod=30
Jul 02 14:07:10.363718 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:10.363680    2555 generic.go:356] "Generic (PLEG): container finished" podID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerID="32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc" exitCode=2
Jul 02 14:07:10.363892 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:10.363756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" event={"ID":"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7","Type":"ContainerDied","Data":"32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc"}
Jul 02 14:07:12.686903 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.686871    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 14:07:12.828838 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.828805    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls\") pod \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") "
Jul 02 14:07:12.828989 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.828860    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-rbac-proxy-sar-config\") pod \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") "
Jul 02 14:07:12.828989 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.828904    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6977w\" (UniqueName: \"kubernetes.io/projected/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-api-access-6977w\") pod \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\" (UID: \"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7\") "
Jul 02 14:07:12.829270 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.829246    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" (UID: "09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:07:12.831080 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.831054    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" (UID: "09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:07:12.831080 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.831059    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-api-access-6977w" (OuterVolumeSpecName: "kube-api-access-6977w") pod "09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" (UID: "09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7"). InnerVolumeSpecName "kube-api-access-6977w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:07:12.929854 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.929821    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 14:07:12.929854 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.929850    2555 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 14:07:12.929854 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:12.929859    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6977w\" (UniqueName: \"kubernetes.io/projected/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7-kube-api-access-6977w\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 14:07:13.210055 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.209959    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:13.378037 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.377998    2555 generic.go:356] "Generic (PLEG): container finished" podID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerID="ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f" exitCode=0
Jul 02 14:07:13.378212 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.378061    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"
Jul 02 14:07:13.378212 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.378063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" event={"ID":"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7","Type":"ContainerDied","Data":"ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f"}
Jul 02 14:07:13.378212 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.378190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk" event={"ID":"09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7","Type":"ContainerDied","Data":"2d7093590fb4a81980e1a653b1f0ca9015abe5484780f509eb9ca02a33a886ed"}
Jul 02 14:07:13.378346 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.378229    2555 scope.go:117] "RemoveContainer" containerID="32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc"
Jul 02 14:07:13.389442 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.389391    2555 scope.go:117] "RemoveContainer" containerID="ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f"
Jul 02 14:07:13.395035 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.395008    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"]
Jul 02 14:07:13.398381 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.398356    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f184-predictor-5c9f6d84c6-srwjk"]
Jul 02 14:07:13.398443 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.398412    2555 scope.go:117] "RemoveContainer" containerID="32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc"
Jul 02 14:07:13.398752 ip-10-0-133-122 kubenswrapper[2555]: E0702 14:07:13.398725    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc\": container with ID starting with 32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc not found: ID does not exist" containerID="32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc"
Jul 02 14:07:13.398820 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.398764    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc"} err="failed to get container status \"32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc\": rpc error: code = NotFound desc = could not find container \"32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc\": container with ID starting with 32b146284f669548ff60dcfa7157a242d9e512cd9a1ce16cca49ab6ddf1999bc not found: ID does not exist"
Jul 02 14:07:13.398820 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.398792    2555 scope.go:117] "RemoveContainer" containerID="ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f"
Jul 02 14:07:13.399050 ip-10-0-133-122 kubenswrapper[2555]: E0702 14:07:13.399029    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f\": container with ID starting with ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f not found: ID does not exist" containerID="ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f"
Jul 02 14:07:13.399092 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:13.399057    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f"} err="failed to get container status \"ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f\": rpc error: code = NotFound desc = could not find container \"ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f\": container with ID starting with ea1ccfbb9caad8fb690b9db27d1a86ef323bf58d4d9ce71512d3b75b65d1343f not found: ID does not exist"
Jul 02 14:07:15.226099 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:15.226004    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" path="/var/lib/kubelet/pods/09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7/volumes"
Jul 02 14:07:18.209577 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:18.209533    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:23.209382 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:23.209346    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:23.209808 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:23.209455    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 14:07:26.380899 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:26.380868    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:27.207785 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:27.207759    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:28.017043 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:28.017011    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:28.209650 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:28.209599    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:28.794823 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:28.794792    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:29.576131 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:29.576101    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:30.368784 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:30.368748    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:31.156038 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:31.156005    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:31.950656 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:31.950623    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:32.726026 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:32.725997    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:33.209382 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:33.209343    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:33.518506 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:33.518426    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:34.312146 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:34.312113    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:35.098649 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:35.098613    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:35.887653 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:35.887621    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:36.712583 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:36.712548    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-7f184-7676f88dd9-z4frx_175b54bb-8dce-4f40-9044-70aef274b1ae/switch-graph-7f184/0.log"
Jul 02 14:07:38.210115 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:38.210081    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:07:39.489148 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.489109    2555 generic.go:356] "Generic (PLEG): container finished" podID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerID="1bfe8c95096d50dbb8f0f9feb31ea571f4e0ff59f5fe386083579f63ba5c1cbd" exitCode=0
Jul 02 14:07:39.489525 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.489179    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" event={"ID":"175b54bb-8dce-4f40-9044-70aef274b1ae","Type":"ContainerDied","Data":"1bfe8c95096d50dbb8f0f9feb31ea571f4e0ff59f5fe386083579f63ba5c1cbd"}
Jul 02 14:07:39.601576 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.601545    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 14:07:39.651355 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.651316    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/175b54bb-8dce-4f40-9044-70aef274b1ae-openshift-service-ca-bundle\") pod \"175b54bb-8dce-4f40-9044-70aef274b1ae\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") "
Jul 02 14:07:39.651355 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.651361    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls\") pod \"175b54bb-8dce-4f40-9044-70aef274b1ae\" (UID: \"175b54bb-8dce-4f40-9044-70aef274b1ae\") "
Jul 02 14:07:39.651686 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.651663    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175b54bb-8dce-4f40-9044-70aef274b1ae-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "175b54bb-8dce-4f40-9044-70aef274b1ae" (UID: "175b54bb-8dce-4f40-9044-70aef274b1ae"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:07:39.653451 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.653424    2555 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "175b54bb-8dce-4f40-9044-70aef274b1ae" (UID: "175b54bb-8dce-4f40-9044-70aef274b1ae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:07:39.752115 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.752006    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/175b54bb-8dce-4f40-9044-70aef274b1ae-openshift-service-ca-bundle\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 14:07:39.752115 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:39.752055    2555 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/175b54bb-8dce-4f40-9044-70aef274b1ae-proxy-tls\") on node \"ip-10-0-133-122.ec2.internal\" DevicePath \"\""
Jul 02 14:07:40.494768 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.494737    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"
Jul 02 14:07:40.494768 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.494746    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx" event={"ID":"175b54bb-8dce-4f40-9044-70aef274b1ae","Type":"ContainerDied","Data":"368cac13b5e262d97f17dac360211c76ec107aeef266f3089f3a87c8278cc8ef"}
Jul 02 14:07:40.495396 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.494810    2555 scope.go:117] "RemoveContainer" containerID="1bfe8c95096d50dbb8f0f9feb31ea571f4e0ff59f5fe386083579f63ba5c1cbd"
Jul 02 14:07:40.514735 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.514687    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"]
Jul 02 14:07:40.516961 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.516934    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-7f184-7676f88dd9-z4frx"]
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.831348    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5gcbj/must-gather-5t6tx"]
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832066    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kube-rbac-proxy"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832085    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kube-rbac-proxy"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832110    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832119    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832129    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832138    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832152    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832168    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832201    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kube-rbac-proxy"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832209    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kube-rbac-proxy"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832233    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832241    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832359    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kserve-container"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832370    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5462ee6-1101-4006-ac12-c3069ae275b7" containerName="splitter-graph-03d0f"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832386    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ac4a1b-aff2-4e55-9f19-87b60d5ad4a7" containerName="kube-rbac-proxy"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832395    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kserve-container"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832406    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b43aeb7-3760-4843-b59b-718ce64e0011" containerName="kube-rbac-proxy"
Jul 02 14:07:40.835978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.832422    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" containerName="switch-graph-7f184"
Jul 02 14:07:40.837484 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.837467    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:40.840272 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.840248    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5gcbj\"/\"kube-root-ca.crt\""
Jul 02 14:07:40.840424 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.840302    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5gcbj\"/\"openshift-service-ca.crt\""
Jul 02 14:07:40.840424 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.840310    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5gcbj\"/\"default-dockercfg-9lfdl\""
Jul 02 14:07:40.840739 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.840697    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gcbj/must-gather-5t6tx"]
Jul 02 14:07:40.963602 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.963566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckkpw\" (UniqueName: \"kubernetes.io/projected/c9766382-ff6c-475a-b283-7368eee8b1c1-kube-api-access-ckkpw\") pod \"must-gather-5t6tx\" (UID: \"c9766382-ff6c-475a-b283-7368eee8b1c1\") " pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:40.963806 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:40.963618    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9766382-ff6c-475a-b283-7368eee8b1c1-must-gather-output\") pod \"must-gather-5t6tx\" (UID: \"c9766382-ff6c-475a-b283-7368eee8b1c1\") " pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:41.064934 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.064889    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckkpw\" (UniqueName: \"kubernetes.io/projected/c9766382-ff6c-475a-b283-7368eee8b1c1-kube-api-access-ckkpw\") pod \"must-gather-5t6tx\" (UID: \"c9766382-ff6c-475a-b283-7368eee8b1c1\") " pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:41.064934 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.064946    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9766382-ff6c-475a-b283-7368eee8b1c1-must-gather-output\") pod \"must-gather-5t6tx\" (UID: \"c9766382-ff6c-475a-b283-7368eee8b1c1\") " pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:41.065244 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.065229    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c9766382-ff6c-475a-b283-7368eee8b1c1-must-gather-output\") pod \"must-gather-5t6tx\" (UID: \"c9766382-ff6c-475a-b283-7368eee8b1c1\") " pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:41.072748 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.072720    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckkpw\" (UniqueName: \"kubernetes.io/projected/c9766382-ff6c-475a-b283-7368eee8b1c1-kube-api-access-ckkpw\") pod \"must-gather-5t6tx\" (UID: \"c9766382-ff6c-475a-b283-7368eee8b1c1\") " pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:41.147064 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.146980    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gcbj/must-gather-5t6tx"
Jul 02 14:07:41.225872 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.225843    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175b54bb-8dce-4f40-9044-70aef274b1ae" path="/var/lib/kubelet/pods/175b54bb-8dce-4f40-9044-70aef274b1ae/volumes"
Jul 02 14:07:41.278388 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.278360    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gcbj/must-gather-5t6tx"]
Jul 02 14:07:41.282056 ip-10-0-133-122 kubenswrapper[2555]: W0702 14:07:41.282029    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9766382_ff6c_475a_b283_7368eee8b1c1.slice/crio-c8c1fc85baa53cf3eacc7087c992c60812bf64b7650a8b860fbfa2d14068dc47 WatchSource:0}: Error finding container c8c1fc85baa53cf3eacc7087c992c60812bf64b7650a8b860fbfa2d14068dc47: Status 404 returned error can't find the container with id c8c1fc85baa53cf3eacc7087c992c60812bf64b7650a8b860fbfa2d14068dc47
Jul 02 14:07:41.283962 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.283939    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:07:41.499978 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:41.499885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gcbj/must-gather-5t6tx" event={"ID":"c9766382-ff6c-475a-b283-7368eee8b1c1","Type":"ContainerStarted","Data":"c8c1fc85baa53cf3eacc7087c992c60812bf64b7650a8b860fbfa2d14068dc47"}
Jul 02 14:07:42.509974 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:42.509890    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gcbj/must-gather-5t6tx" event={"ID":"c9766382-ff6c-475a-b283-7368eee8b1c1","Type":"ContainerStarted","Data":"4c6c7c79f7835b4fea37a1e876af4f2ccc923807a9b13897afffb74210ff88c6"}
Jul 02 14:07:43.520182 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:43.519572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gcbj/must-gather-5t6tx" event={"ID":"c9766382-ff6c-475a-b283-7368eee8b1c1","Type":"ContainerStarted","Data":"dcefb6810898b7665103b4f2d3e89e751de204921d3b550770815b746adcb4f9"}
Jul 02 14:07:43.536676 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:43.536618    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5gcbj/must-gather-5t6tx" podStartSLOduration=2.470298956 podStartE2EDuration="3.536598438s" totalImagesPullingTime="1.066299482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:07:43.534256861 +0000 UTC m=+3170.724586533" watchObservedRunningTime="2026-07-02 14:07:43.536598438 +0000 UTC m=+3170.726928110"
Jul 02 14:07:43.580912 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:43.580874    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5b2xr_9b39b0b0-9e86-4268-8309-fca12d165a04/global-pull-secret-syncer/0.log"
Jul 02 14:07:43.749220 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:43.749193    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gmn8c_6d64dd38-cdb4-4b8c-a31e-39ff013ff70f/konnectivity-agent/0.log"
Jul 02 14:07:43.786157 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:43.786081    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-122.ec2.internal_cf6c904bbcfca3cc936c3769d2fdfa74/haproxy/0.log"
Jul 02 14:07:48.637385 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.637353    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-j2xzr_01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f/kube-state-metrics/0.log"
Jul 02 14:07:48.657595 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.657569    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-j2xzr_01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f/kube-rbac-proxy-main/0.log"
Jul 02 14:07:48.675543 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.675510    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-j2xzr_01fa2353-6942-4c5f-b3d8-7e6e6cbbb64f/kube-rbac-proxy-self/0.log"
Jul 02 14:07:48.700139 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.700092    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d6d8c96cf-zl9xk_3269a7ce-84b9-4f90-99e4-7c2730e9b336/metrics-server/0.log"
Jul 02 14:07:48.750199 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.750163    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-96v2m_9c24256d-7842-44b0-9f28-153d5e5c1176/node-exporter/0.log"
Jul 02 14:07:48.769037 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.769002    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-96v2m_9c24256d-7842-44b0-9f28-153d5e5c1176/kube-rbac-proxy/0.log"
Jul 02 14:07:48.792868 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.792840    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-96v2m_9c24256d-7842-44b0-9f28-153d5e5c1176/init-textfile/0.log"
Jul 02 14:07:48.952981 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.952900    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-tvw9b_4d17e46d-255d-4497-b605-445ed68e4cc2/kube-rbac-proxy-main/0.log"
Jul 02 14:07:48.970664 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.970636    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-tvw9b_4d17e46d-255d-4497-b605-445ed68e4cc2/kube-rbac-proxy-self/0.log"
Jul 02 14:07:48.992792 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:48.992753    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-57f99c8b49-tvw9b_4d17e46d-255d-4497-b605-445ed68e4cc2/openshift-state-metrics/0.log"
Jul 02 14:07:51.099601 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:51.099571    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/1.log"
Jul 02 14:07:51.111619 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:51.111584    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-mcdfs_58538733-ded0-48b6-b964-00c1fa92b611/console-operator/2.log"
Jul 02 14:07:51.497360 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:51.497278    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c55ff46-xg8h9_db5d3855-ec4b-4fdb-abf4-dbc131caa490/console/0.log"
Jul 02 14:07:51.538620 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:51.538591    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-lcvtp_2d108038-2488-4d4c-917c-3e88960887d3/download-server/0.log"
Jul 02 14:07:52.060131 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.060086    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"]
Jul 02 14:07:52.066042 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.066014    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.072423 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.072393    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"]
Jul 02 14:07:52.173729 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.173676    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzb7l\" (UniqueName: \"kubernetes.io/projected/e4bcc918-af64-46fe-b8db-14c3853fa66b-kube-api-access-hzb7l\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.174216 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.173739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-lib-modules\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.174216 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.173786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-sys\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.174216 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.173846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-proc\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.174216 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.173893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-podres\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275153 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzb7l\" (UniqueName: \"kubernetes.io/projected/e4bcc918-af64-46fe-b8db-14c3853fa66b-kube-api-access-hzb7l\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275153 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275155    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-lib-modules\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275393 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275184    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-sys\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275393 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-proc\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275393 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275266    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-podres\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275393 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275325    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-lib-modules\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275393 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275331    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-sys\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275393 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275358    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-podres\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.275393 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.275338    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e4bcc918-af64-46fe-b8db-14c3853fa66b-proc\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.283893 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.283866    2555 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzb7l\" (UniqueName: \"kubernetes.io/projected/e4bcc918-af64-46fe-b8db-14c3853fa66b-kube-api-access-hzb7l\") pod \"perf-node-gather-daemonset-thhxc\" (UID: \"e4bcc918-af64-46fe-b8db-14c3853fa66b\") " pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.381647 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.381620    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:52.561448 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.561369    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"]
Jul 02 14:07:52.564370 ip-10-0-133-122 kubenswrapper[2555]: W0702 14:07:52.564166    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode4bcc918_af64_46fe_b8db_14c3853fa66b.slice/crio-6be22efa6e7b7132f36ddeed92aa2e2b105a4c5388aa3dc170a40d2d6c0a6fa0 WatchSource:0}: Error finding container 6be22efa6e7b7132f36ddeed92aa2e2b105a4c5388aa3dc170a40d2d6c0a6fa0: Status 404 returned error can't find the container with id 6be22efa6e7b7132f36ddeed92aa2e2b105a4c5388aa3dc170a40d2d6c0a6fa0
Jul 02 14:07:52.578365 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.575988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc" event={"ID":"e4bcc918-af64-46fe-b8db-14c3853fa66b","Type":"ContainerStarted","Data":"6be22efa6e7b7132f36ddeed92aa2e2b105a4c5388aa3dc170a40d2d6c0a6fa0"}
Jul 02 14:07:52.878104 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.878069    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dl649_fc8cbf88-1d0a-4f50-afb3-2b828a2902f8/dns/0.log"
Jul 02 14:07:52.895354 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.895313    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dl649_fc8cbf88-1d0a-4f50-afb3-2b828a2902f8/kube-rbac-proxy/0.log"
Jul 02 14:07:52.989279 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:52.989201    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m5qlt_68762dfb-70dc-4f7c-a38b-6a501e0c52a6/dns-node-resolver/0.log"
Jul 02 14:07:53.536357 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:53.536321    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zvw8r_db12a0b5-31c6-45dc-8dc3-0bf8e847e7f1/node-ca/0.log"
Jul 02 14:07:53.584990 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:53.584936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc" event={"ID":"e4bcc918-af64-46fe-b8db-14c3853fa66b","Type":"ContainerStarted","Data":"e01050161645b066b2ef1f50318f73af3db2045a52b01f77e99c241cd67552f1"}
Jul 02 14:07:53.586012 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:53.585733    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:07:53.599977 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:53.599905    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc" podStartSLOduration=1.599885045 podStartE2EDuration="1.599885045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:07:53.599272969 +0000 UTC m=+3180.789602642" watchObservedRunningTime="2026-07-02 14:07:53.599885045 +0000 UTC m=+3180.790214717"
Jul 02 14:07:54.271027 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:54.270995    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64b958756c-vc5td_27cf9350-a187-4386-bd48-3e92edd6810d/router/0.log"
Jul 02 14:07:54.618753 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:54.618686    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8q44h_7119527a-4788-4c12-9ca1-198187511679/serve-healthcheck-canary/0.log"
Jul 02 14:07:55.054977 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:55.054901    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9jmz2_7b0558b8-7007-427c-b340-f220628ec8d9/kube-rbac-proxy/0.log"
Jul 02 14:07:55.071456 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:55.071432    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9jmz2_7b0558b8-7007-427c-b340-f220628ec8d9/exporter/0.log"
Jul 02 14:07:55.088669 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:55.088637    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9jmz2_7b0558b8-7007-427c-b340-f220628ec8d9/extractor/0.log"
Jul 02 14:07:57.160403 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:57.160354    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-694f5b7594-9n2bc_5ac594ed-8060-4a56-b0c1-e08b7f70085b/manager/0.log"
Jul 02 14:07:57.178608 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:57.178583    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-9vkdn_e09a3183-0bb6-4744-be2b-ecf1df64fa4c/manager/0.log"
Jul 02 14:07:59.602772 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:07:59.602744    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5gcbj/perf-node-gather-daemonset-thhxc"
Jul 02 14:08:01.357698 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:01.357656    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-vszvw_17c14389-121d-446e-b6a5-dd612b23c463/kube-storage-version-migrator-operator/1.log"
Jul 02 14:08:01.361196 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:01.361165    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-vszvw_17c14389-121d-446e-b6a5-dd612b23c463/kube-storage-version-migrator-operator/0.log"
Jul 02 14:08:02.433374 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.433299    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c75cn_26b8bcfb-8adf-4773-ae67-670e9899f5bb/kube-multus-additional-cni-plugins/0.log"
Jul 02 14:08:02.455882 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.455849    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c75cn_26b8bcfb-8adf-4773-ae67-670e9899f5bb/egress-router-binary-copy/0.log"
Jul 02 14:08:02.474210 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.474182    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c75cn_26b8bcfb-8adf-4773-ae67-670e9899f5bb/cni-plugins/0.log"
Jul 02 14:08:02.505487 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.505445    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c75cn_26b8bcfb-8adf-4773-ae67-670e9899f5bb/bond-cni-plugin/0.log"
Jul 02 14:08:02.522363 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.522335    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c75cn_26b8bcfb-8adf-4773-ae67-670e9899f5bb/routeoverride-cni/0.log"
Jul 02 14:08:02.539662 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.539636    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c75cn_26b8bcfb-8adf-4773-ae67-670e9899f5bb/whereabouts-cni-bincopy/0.log"
Jul 02 14:08:02.557363 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.557285    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c75cn_26b8bcfb-8adf-4773-ae67-670e9899f5bb/whereabouts-cni/0.log"
Jul 02 14:08:02.778824 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.778737    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ngm5d_484a8f20-725a-4f94-a670-a40f0db84f7c/kube-multus/0.log"
Jul 02 14:08:02.887436 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.887388    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p2b6h_7c51b08e-60d3-4455-b38a-74bee9c35ebf/network-metrics-daemon/0.log"
Jul 02 14:08:02.905301 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:02.905271    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p2b6h_7c51b08e-60d3-4455-b38a-74bee9c35ebf/kube-rbac-proxy/0.log"
Jul 02 14:08:03.842795 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.842769    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-controller/0.log"
Jul 02 14:08:03.855522 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.855495    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/0.log"
Jul 02 14:08:03.855669 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.855529    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovn-acl-logging/1.log"
Jul 02 14:08:03.876887 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.876864    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/kube-rbac-proxy-node/0.log"
Jul 02 14:08:03.898098 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.898070    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 14:08:03.911649 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.911625    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/northd/0.log"
Jul 02 14:08:03.927455 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.927429    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/nbdb/0.log"
Jul 02 14:08:03.945652 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:03.945633    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/sbdb/0.log"
Jul 02 14:08:04.012140 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:04.012066    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kg5l8_16bba2ca-171b-41b4-91c4-d120744ecce2/ovnkube-controller/0.log"
Jul 02 14:08:05.078754 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:05.078698    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-f9hzk_a8a20105-d33b-44c0-a6fa-9407191d1dd4/network-check-target-container/0.log"
Jul 02 14:08:05.977872 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:05.977841    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fzx5l_70e3cf95-2e8f-4c7c-9a7f-2ac59b982846/iptables-alerter/0.log"
Jul 02 14:08:06.640742 ip-10-0-133-122 kubenswrapper[2555]: I0702 14:08:06.640714    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-w2h44_736527e9-081f-423d-9b9e-2a38ab79f52a/tuned/0.log"