Jul 01 19:19:08.348562 ip-10-0-130-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 19:19:08.348576 ip-10-0-130-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 19:19:08.348584 ip-10-0-130-234 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 19:19:08.348868 ip-10-0-130-234 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 19:19:18.574113 ip-10-0-130-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 19:19:18.574134 ip-10-0-130-234 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 84daf9112f184bb79c7ef44838290bc5 --
Jul 01 19:21:11.628333 ip-10-0-130-234 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 19:21:12.057739 ip-10-0-130-234 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:12.057739 ip-10-0-130-234 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 19:21:12.057739 ip-10-0-130-234 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:12.057739 ip-10-0-130-234 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 19:21:12.057739 ip-10-0-130-234 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 19:21:12.057739 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.054471    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060523    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060535    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060539    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060542    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060546    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060549    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060551    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060554    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060557    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060559    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060567    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060570    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060572    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060575    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060578    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060581    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060583    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060586    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060589    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:12.062861 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060591    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060594    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060596    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060599    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060601    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060604    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060607    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060610    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060613    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060615    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060619    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060622    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060625    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060628    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060630    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060633    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060636    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060639    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060641    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060644    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:12.066108 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060647    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060649    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060651    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060654    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060656    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060659    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060661    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060664    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060666    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060669    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060672    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060674    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060678    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060680    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060683    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060685    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060688    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060691    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060693    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060696    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:12.068511 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060698    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060700    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060704    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060708    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060711    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060714    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060716    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060719    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060722    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060724    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060727    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060729    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060733    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060735    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060738    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060740    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060743    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060745    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060748    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060750    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:12.071541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060754    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060756    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060761    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060764    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060766    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060769    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060771    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060774    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060776    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060779    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060781    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060784    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060786    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060789    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060791    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060793    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060797    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060799    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060803    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:12.073887 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.060807    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061635    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061643    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061647    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061650    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061654    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061657    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061660    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061662    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061665    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061668    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061670    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061673    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061677    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061679    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061682    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061685    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061687    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061690    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:12.077198 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061693    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061695    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061698    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061701    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061703    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061707    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061711    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061714    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061717    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061719    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061722    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061725    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061727    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061730    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061732    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061735    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061738    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061740    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061743    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061746    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:12.112780 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061750    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061752    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061755    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061757    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061759    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061762    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061765    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061767    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061769    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061772    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061774    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061777    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061779    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061781    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061784    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061787    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061789    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061792    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061796    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061799    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:12.114323 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061802    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061805    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061807    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061810    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061812    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061815    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061820    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061822    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061825    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061827    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061830    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061833    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061835    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061839    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061842    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061845    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061849    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061852    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061855    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:12.122431 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061857    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061860    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061863    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061865    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061868    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061870    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061872    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061875    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061877    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061880    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061882    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061885    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061887    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061890    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061892    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061894    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061897    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061899    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061901    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061906    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:12.132559 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061908    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.061911    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062335    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062346    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062349    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062352    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062355    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062358    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062361    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062364    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062367    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062369    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062372    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062375    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062377    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062380    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062382    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062385    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062387    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062390    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:12.151737 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062393    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:12.143669 ip-10-0-130-234 systemd[1]: Started Kubernetes Kubelet.
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062395    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062398    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062401    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062403    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062406    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062409    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062413    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062416    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062419    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062422    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062424    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062426    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062430    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062433    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062436    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062439    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062442    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062445    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062448    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:12.165393 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062450    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062453    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062456    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062458    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062462    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062464    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062467    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062470    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062472    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062475    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062477    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062480    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062483    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062486    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062489    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062491    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062494    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062498    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062502    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:12.378085 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062504    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062507    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062510    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062512    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062524    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062528    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062531    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062534    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062537    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062540    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062543    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062545    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062548    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062550    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062553    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062555    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062558    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062560    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062563    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:12.396968 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062565    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062568    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062573    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062576    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062578    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062580    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062583    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062585    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062588    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062590    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062593    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062596    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062598    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062601    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062603    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062606    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062608    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062611    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062613    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062616    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:12.398704 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062618    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062621    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.062629    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062697    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062704    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062718    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062722    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062726    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062729    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062733    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062737    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062740    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062743    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062746    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062751    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062754    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062757    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062760    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062763    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062765    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062768    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062775    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062778    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 19:21:12.399818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062781    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062784    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062787    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062791    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062794    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062796    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062800    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062803    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062806    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062809    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062812    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062815    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062826    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062830    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062833    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062836    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062839    2576 flags.go:64] FLAG: --enable-server="true"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062842    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062849    2576 flags.go:64] FLAG: --event-burst="100"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062852    2576 flags.go:64] FLAG: --event-qps="50"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062855    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062858    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062861    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062868    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 19:21:12.401338 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062872    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062875    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062878    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062883    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062886    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062888    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062891    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062894    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062897    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062900    2576 flags.go:64] FLAG: --feature-gates=""
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062903    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062906    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062909    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062912    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062915    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062918    2576 flags.go:64] FLAG: --help="false"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062920    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062923    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062926    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062929    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062932    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062936    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062939    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 19:21:12.405829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062942    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062945    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062948    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062950    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062953    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062956    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062959    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062962    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062966    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062969    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062972    2576 flags.go:64] FLAG: --lock-file=""
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062974    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062978    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.062981    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063006    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063009    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063012    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063015    2576 flags.go:64] FLAG: --logging-format="text"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063018    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063021    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063024    2576 flags.go:64] FLAG: --manifest-url=""
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063026    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063031    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063034    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063038    2576 flags.go:64] FLAG: --max-pods="110"
Jul 01 19:21:12.406923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063040    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063043    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063046    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063049    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063052    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063054    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063058    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063064    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063067    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063070    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063073    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063076    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063081    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063084    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063087    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063089    2576 flags.go:64] FLAG: --port="10250"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063094    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063097    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08791c2a411c3da3d"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063100    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063104    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063106    2576 flags.go:64] FLAG: --register-node="true"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063109    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063113    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063116    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 01 19:21:12.408240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063118    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063121    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063125    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063128    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063131    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063134    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063137    2576 flags.go:64] FLAG: --runonce="false"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063139    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063142    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063146    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063148    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063151    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063154    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063157    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063160    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063166    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063169    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063172    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063175    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063178    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063181    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063183    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063189    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063191    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063196    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 19:21:12.409966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063202    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063205    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063209    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063212    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063215    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063218    2576 flags.go:64] FLAG: --v="2"
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063221    2576 flags.go:64] FLAG: --version="false"
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063225    2576 flags.go:64] FLAG: --vmodule=""
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063229    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.063232    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063343    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063347    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063351    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063354    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063356    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063359    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063362    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063364    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063367    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063370    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063372    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:12.411489 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063375    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063377    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063387    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063391    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063395    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063399    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063402    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063405    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063408    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063410    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063414    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063417    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063419    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063423    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063426    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063428    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063431    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063433    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063436    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063439    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:12.412603 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063441    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063444    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063446    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063449    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063451    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063454    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063457    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063461    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063464    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063466    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063470    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063472    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063475    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063478    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063480    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063488    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063492    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063495    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063497    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063500    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:12.415022 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063502    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063505    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063509    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063511    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063514    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063517    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063520    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063523    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063525    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063528    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063530    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063533    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063535    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063538    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063541    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063544    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063546    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063549    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063551    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063554    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:12.416295 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063556    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063559    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063562    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063564    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063567    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063569    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063571    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063574    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063577    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063580    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063582    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063585    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063587    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063590    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063594    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063596    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063599    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063603    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063606    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063608    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:12.417756 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063611    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063613    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063616    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063619    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063621    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063624    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063626    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.063629    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.064389    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.071317    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.071331    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.071343    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.071351    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071404    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071408    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:12.418635 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071412    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071415    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071418    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071421    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071424    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071427    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071430    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071432    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071435    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071438    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071442    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071446    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071448    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071451    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071454    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071456    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071459    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071462    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071465    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:12.419935 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071467    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071470    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071472    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071475    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071477    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071480    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071482    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071485    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071488    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071491    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071493    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071496    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071498    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071501    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071504    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071506    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071508    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071511    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071514    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071517    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:12.420791 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071519    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071521    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071524    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071526    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071529    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071531    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071534    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071536    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071539    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071541    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071544    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071548    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071552    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071555    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071558    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071561    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071563    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071566    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071569    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:12.421660 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071571    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071575    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071578    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071581    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071583    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071586    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071589    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071591    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071593    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071596    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071598    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071601    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071603    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071607    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071609    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071612    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071614    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071617    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071619    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071622    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:12.422509 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071625    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071627    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071629    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071632    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071634    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071637    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071640    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071642    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071645    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071648    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071650    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071653    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071656    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071659    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071662    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071665    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071668    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071671    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:12.423399 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071675    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.071680    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071800    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071804    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071807    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071811    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071813    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071816    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071820    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071823    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071825    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071828    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071830    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071833    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071835    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 19:21:12.424282 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071838    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071840    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071843    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071846    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071848    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071851    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071853    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071856    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071858    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071861    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071863    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071866    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071868    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071871    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071874    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071876    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071879    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071881    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071884    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071886    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 19:21:12.424943 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071889    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071891    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071894    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071897    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071899    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071902    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071905    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071907    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071909    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071912    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071915    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071917    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071920    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071922    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071926    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071930    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071933    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071936    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071939    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071941    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 19:21:12.425793 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071944    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071946    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071949    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071951    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071953    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071957    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071959    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071964    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071966    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071969    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071971    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071974    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071977    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071979    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071995    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.071999    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072002    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072004    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072007    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 19:21:12.426672 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072012    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072015    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072019    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072022    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072025    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072028    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072030    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072033    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072035    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072038    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072040    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072043    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072045    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072048    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072050    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072052    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072055    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072057    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072060    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072062    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 19:21:12.427541 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072065    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072068    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072070    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072073    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072076    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072078    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:12.072081    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.072086    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.072178    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.076318    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.078004    2576 server.go:1019] "Starting client certificate rotation"
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.078106    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.078148    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 19:21:12.428442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.081107    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:21:12.429105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.098205    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 19:21:12.429105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.106749    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 01 19:21:12.429105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.110658    2576 log.go:25] "Validated CRI v1 image API"
Jul 01 19:21:12.429105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.114185    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 19:21:12.429105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.116929    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 89c5ca15-975b-4f0e-95ea-d0d1ab0cd569:/dev/nvme0n1p4 f61c014b-6142-4017-b8ee-2c77c82f4838:/dev/nvme0n1p3]
Jul 01 19:21:12.429105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.116955    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 19:21:12.429105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.123028    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.123320    2576 manager.go:216] Machine: {Timestamp:2026-07-01 19:21:12.1215331 +0000 UTC m=+0.387523807 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097204 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bf70b2d2e0d1ff3474fb880d56e30 SystemUUID:ec2bf70b-2d2e-0d1f-f347-4fb880d56e30 BootID:84daf911-2f18-4bb7-9c7e-f44838290bc5 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:9c:af:43:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:9c:af:43:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:70:3e:f0:d9:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.123426    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.123523    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.126845    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.126867    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-234.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.127705    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.127714    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.127722    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.128535    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.129750    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134332    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134447    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134470    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134486    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134496    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134504    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134513    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 19:21:12.430434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134762    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.134777    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.137546    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139191    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139218    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139361    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139372    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139378    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139384    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139398    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.139404    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140059    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140065    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140072    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140077    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140094    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140103    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140122    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.140128    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.142963    2576 server.go:1268] "Started kubelet"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.143106    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.143173    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.143162    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.147152    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.148897    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.151704    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-234.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.151751    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.151748    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.152371    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.157304    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.157315    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.157380    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158024    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158026    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158051    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158099    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158112    2576 factory.go:55] Registering systemd factory
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158120    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158140    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158148    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158311    2576 factory.go:153] Registering CRI-O factory
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158340    2576 factory.go:223] Registration of the crio container factory successfully
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158362    2576 factory.go:103] Registering Raw factory
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.158375    2576 manager.go:1196] Started watching for new ooms in manager
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.159448    2576 manager.go:349] Starting recovery of all containers
Jul 01 19:21:12.432361 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.163341    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.163442    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-234.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.163485    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-234.ec2.internal.18be40ba7a29a19a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-234.ec2.internal,UID:ip-10-0-130-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-234.ec2.internal,},FirstTimestamp:2026-07-01 19:21:12.142938522 +0000 UTC m=+0.408929227,LastTimestamp:2026-07-01 19:21:12.142938522 +0000 UTC m=+0.408929227,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-234.ec2.internal,}"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.170949    2576 manager.go:354] Recovery completed
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.175425    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.178183    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.178217    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.178227    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.178697    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.178706    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.178708    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.178721    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.179788    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.179817    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.179830    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.181561    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-234.ec2.internal.18be40ba7c43b1c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-234.ec2.internal,UID:ip-10-0-130-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-234.ec2.internal,},FirstTimestamp:2026-07-01 19:21:12.178201032 +0000 UTC m=+0.444191739,LastTimestamp:2026-07-01 19:21:12.178201032 +0000 UTC m=+0.444191739,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-234.ec2.internal,}"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.183794    2576 policy_none.go:49] "None policy: Start"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.183806    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.183816    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.185774    2576 policy_none.go:47] "Start"
Jul 01 19:21:12.434337 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.190509    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-234.ec2.internal.18be40ba7c440318  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-234.ec2.internal,UID:ip-10-0-130-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-234.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-234.ec2.internal,},FirstTimestamp:2026-07-01 19:21:12.178221848 +0000 UTC m=+0.444212554,LastTimestamp:2026-07-01 19:21:12.178221848 +0000 UTC m=+0.444212554,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-234.ec2.internal,}"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.201794    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-234.ec2.internal.18be40ba7c4426c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-234.ec2.internal,UID:ip-10-0-130-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-234.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-234.ec2.internal,},FirstTimestamp:2026-07-01 19:21:12.178230985 +0000 UTC m=+0.444221691,LastTimestamp:2026-07-01 19:21:12.178230985 +0000 UTC m=+0.444221691,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-234.ec2.internal,}"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.202159    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7fnkh"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.210077    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7fnkh"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.214349    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-234.ec2.internal.18be40ba7c43b1c8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-234.ec2.internal.18be40ba7c43b1c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-234.ec2.internal,UID:ip-10-0-130-234.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-234.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-234.ec2.internal,},FirstTimestamp:2026-07-01 19:21:12.178201032 +0000 UTC m=+0.444191739,LastTimestamp:2026-07-01 19:21:12.179800456 +0000 UTC m=+0.445791167,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-234.ec2.internal,}"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.218522    2576 manager.go:337] "Starting Device Plugin manager"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.218547    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.218555    2576 server.go:87] "Starting device plugin registration server"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.218757    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.218769    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.218851    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.218918    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.218927    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.221657    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.221703    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.236529    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.237674    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.237703    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.237719    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.237748    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.240105    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.319316    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.320112    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.320140    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.320154    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.320182    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.329216    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.329232    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-234.ec2.internal\": node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.338541    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"]
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.338598    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.339296    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:12.580719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.339319    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.339328    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.341505    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.341629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.341652    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.342146    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.342167    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.342149    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.342179    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.342198    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.342210    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.344277    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.344297    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.344873    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.344898    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.344909    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeHasSufficientPID"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.357318    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.359423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24c23ff45096c6e80b4ac6b382d5677d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-234.ec2.internal\" (UID: \"24c23ff45096c6e80b4ac6b382d5677d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.359443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1c28a8c9ff7049a9f9a806acf5cb4eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal\" (UID: \"c1c28a8c9ff7049a9f9a806acf5cb4eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.359462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1c28a8c9ff7049a9f9a806acf5cb4eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal\" (UID: \"c1c28a8c9ff7049a9f9a806acf5cb4eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.384013    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-234.ec2.internal\" not found" node="ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.388334    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-234.ec2.internal\" not found" node="ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.458283    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.460444    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24c23ff45096c6e80b4ac6b382d5677d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-234.ec2.internal\" (UID: \"24c23ff45096c6e80b4ac6b382d5677d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.460466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1c28a8c9ff7049a9f9a806acf5cb4eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal\" (UID: \"c1c28a8c9ff7049a9f9a806acf5cb4eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.460483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1c28a8c9ff7049a9f9a806acf5cb4eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal\" (UID: \"c1c28a8c9ff7049a9f9a806acf5cb4eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.582156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.460519    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1c28a8c9ff7049a9f9a806acf5cb4eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal\" (UID: \"c1c28a8c9ff7049a9f9a806acf5cb4eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.583439 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.460577    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24c23ff45096c6e80b4ac6b382d5677d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-234.ec2.internal\" (UID: \"24c23ff45096c6e80b4ac6b382d5677d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.583439 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.460601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c1c28a8c9ff7049a9f9a806acf5cb4eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal\" (UID: \"c1c28a8c9ff7049a9f9a806acf5cb4eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.583439 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.558690    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:12.724331 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.659700    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:12.724331 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.687961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:12.724331 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:12.691502    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:14.353494 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:12.759890    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:14.531359 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.366785    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:14.531359 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.467333    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.567833    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-234.ec2.internal\" not found"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.583053    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.583192    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.631026    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.642253    2576 apiserver.go:52] "Watching apiserver"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.652764    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.653142    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-bf587","openshift-ovn-kubernetes/ovnkube-node-sxlfd","kube-system/konnectivity-agent-vxzzn","openshift-cluster-node-tuning-operator/tuned-nbqv4","openshift-image-registry/node-ca-gc59t","openshift-multus/multus-brnls","openshift-multus/network-metrics-daemon-r2hbh","openshift-network-diagnostics/network-check-target-q4qsh","kube-system/global-pull-secret-syncer-nfh7p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5","openshift-dns/node-resolver-mg6xr","openshift-multus/multus-additional-cni-plugins-7hs59"]
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.655673    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.659039    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rtv6f\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.659048    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.659052    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.659066    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.659125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.661208    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.661438    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.661621    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5cbjg\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.661669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.661750    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.664155    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.664166    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.664227    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.664267    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.666286    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.668345    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.668740    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-frpb9\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.669055    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.669254    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.669388    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.669545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.669877    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.693171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqg6\" (UniqueName: \"kubernetes.io/projected/af0058fb-09cb-4671-b2a3-1143bb567f59-kube-api-access-7hqg6\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6-agent-certs\") pod \"konnectivity-agent-vxzzn\" (UID: \"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6\") " pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-system-cni-dir\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-cnibin\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670582    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-cni-binary-copy\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af0058fb-09cb-4671-b2a3-1143bb567f59-host-slash\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af0058fb-09cb-4671-b2a3-1143bb567f59-iptables-alerter-script\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-os-release\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9p8p\" (UniqueName: \"kubernetes.io/projected/14f2a6f4-c9eb-4462-b413-3fef82d07445-kube-api-access-q9p8p\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6-konnectivity-ca\") pod \"konnectivity-agent-vxzzn\" (UID: \"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6\") " pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5qg\" (UniqueName: \"kubernetes.io/projected/a22a2075-444a-40ee-8406-7cd45cd55403-kube-api-access-9f5qg\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.670782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.671261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.694710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.672203    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.672317    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.672494    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.672791    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smxx9\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.672981    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.673256    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.673379    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.673599    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9szs4\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.673805    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.673933    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.674052    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.676541    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9wm8s\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.676596    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.676620    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.676827    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.676901    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.679383    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.679427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.679518    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n8x4c\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.679511    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.681144    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.682308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.682618    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.682675    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.684695    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6ksd6\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.684986    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal"]
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.685049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.685573    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.685576    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.687659    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.687674    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n52r5\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.687689    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.687766    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.692163    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 19:21:14.695780 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.692171    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal"]
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.709635    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vtxgb"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.715957    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vtxgb"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.722232    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 19:16:13 +0000 UTC" deadline="2028-02-29 05:24:31.241644709 +0000 UTC"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.722267    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14578h3m16.519379944s"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.765110    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h44dt\" (UniqueName: \"kubernetes.io/projected/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-kube-api-access-h44dt\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af0058fb-09cb-4671-b2a3-1143bb567f59-iptables-alerter-script\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9p8p\" (UniqueName: \"kubernetes.io/projected/14f2a6f4-c9eb-4462-b413-3fef82d07445-kube-api-access-q9p8p\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-systemd-units\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-systemd\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5qg\" (UniqueName: \"kubernetes.io/projected/a22a2075-444a-40ee-8406-7cd45cd55403-kube-api-access-9f5qg\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-slash\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-run-netns\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-log-socket\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsjmn\" (UniqueName: \"kubernetes.io/projected/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-kube-api-access-zsjmn\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.806720 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-cni-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-multus-certs\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.771985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-system-cni-dir\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-cnibin\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-modprobe-d\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-system-cni-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-etc-kubernetes\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlpk\" (UniqueName: \"kubernetes.io/projected/6ef721b2-130c-4f92-824b-5b9039cee361-kube-api-access-7vlpk\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.772143    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.772208    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:15.272187171 +0000 UTC m=+2.031713897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-var-lib-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysctl-d\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-registration-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dxkv\" (UniqueName: \"kubernetes.io/projected/cb7ee79e-263e-4f31-a115-9374f163a704-kube-api-access-7dxkv\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-cni-bin\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-conf-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.809250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772476    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-system-cni-dir\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-kubelet\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772245    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af0058fb-09cb-4671-b2a3-1143bb567f59-iptables-alerter-script\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-cni-netd\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-run\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-cnibin\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovn-node-metrics-cert\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-device-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-netns\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-hostroot\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ef721b2-130c-4f92-824b-5b9039cee361-multus-daemon-config\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-kubelet-config\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqg6\" (UniqueName: \"kubernetes.io/projected/af0058fb-09cb-4671-b2a3-1143bb567f59-kube-api-access-7hqg6\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:14.810078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-node-log\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp2r\" (UniqueName: \"kubernetes.io/projected/a5710820-7283-4fcf-b3a5-b0291c8f274f-kube-api-access-zfp2r\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-sys\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.772924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j54gs\" (UniqueName: \"kubernetes.io/projected/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-kube-api-access-j54gs\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-dbus\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af0058fb-09cb-4671-b2a3-1143bb567f59-host-slash\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-ovn\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773106    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-socket-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af0058fb-09cb-4671-b2a3-1143bb567f59-host-slash\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7ee79e-263e-4f31-a115-9374f163a704-hosts-file\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-k8s-cni-cncf-io\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-tmp\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-os-release\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-etc-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysctl-conf\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-os-release\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.811938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-cnibin\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-os-release\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-host\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6-konnectivity-ca\") pod \"konnectivity-agent-vxzzn\" (UID: \"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6\") " pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-kubernetes\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6-agent-certs\") pod \"konnectivity-agent-vxzzn\" (UID: \"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6\") " pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-cni-binary-copy\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-env-overrides\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-var-lib-kubelet\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-tuned\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-lib-modules\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-sys-fs\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-cni-bin\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovnkube-config\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773766    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 19:21:14.812784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovnkube-script-lib\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-socket-dir-parent\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef721b2-130c-4f92-824b-5b9039cee361-cni-binary-copy\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-cni-multus\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.773981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysconfig\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-host\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-serviceca\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-kubelet\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-systemd\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774104    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14f2a6f4-c9eb-4462-b413-3fef82d07445-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7ee79e-263e-4f31-a115-9374f163a704-tmp-dir\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774142    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-cni-binary-copy\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774161    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.774588    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14f2a6f4-c9eb-4462-b413-3fef82d07445-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.775607    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6-konnectivity-ca\") pod \"konnectivity-agent-vxzzn\" (UID: \"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6\") " pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:14.814019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.779294    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6-agent-certs\") pod \"konnectivity-agent-vxzzn\" (UID: \"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6\") " pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:14.814670 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.781859    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9p8p\" (UniqueName: \"kubernetes.io/projected/14f2a6f4-c9eb-4462-b413-3fef82d07445-kube-api-access-q9p8p\") pod \"multus-additional-cni-plugins-7hs59\" (UID: \"14f2a6f4-c9eb-4462-b413-3fef82d07445\") " pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:14.814670 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.782046    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:14.814670 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.782063    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:14.814670 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.782075    2576 projected.go:196] Error preparing data for projected volume kube-api-access-b96vl for pod openshift-network-diagnostics/network-check-target-q4qsh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:14.814670 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.782123    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl podName:1e5dcab4-31fb-4c7f-a51e-5dc3a0792059 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:15.282109304 +0000 UTC m=+2.041636018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b96vl" (UniqueName: "kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl") pod "network-check-target-q4qsh" (UID: "1e5dcab4-31fb-4c7f-a51e-5dc3a0792059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:14.814670 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.784244    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5qg\" (UniqueName: \"kubernetes.io/projected/a22a2075-444a-40ee-8406-7cd45cd55403-kube-api-access-9f5qg\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:14.814670 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.785975    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqg6\" (UniqueName: \"kubernetes.io/projected/af0058fb-09cb-4671-b2a3-1143bb567f59-kube-api-access-7hqg6\") pod \"iptables-alerter-bf587\" (UID: \"af0058fb-09cb-4671-b2a3-1143bb567f59\") " pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-ovn\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-socket-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7ee79e-263e-4f31-a115-9374f163a704-hosts-file\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-k8s-cni-cncf-io\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-tmp\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-etc-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysctl-conf\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874470    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7ee79e-263e-4f31-a115-9374f163a704-hosts-file\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-socket-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-ovn\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-cnibin\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874540    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-etc-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-os-release\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874565    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-k8s-cni-cncf-io\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-host\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:14.929174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-cnibin\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-kubernetes\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874615    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-os-release\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874610    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-host\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-env-overrides\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874647    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysctl-conf\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-kubernetes\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-var-lib-kubelet\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-tuned\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-lib-modules\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874744    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-var-lib-kubelet\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-sys-fs\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-run-ovn-kubernetes\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-cni-bin\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-sys-fs\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovnkube-config\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovnkube-script-lib\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-lib-modules\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874863    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-cni-bin\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874878    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-socket-dir-parent\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef721b2-130c-4f92-824b-5b9039cee361-cni-binary-copy\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-cni-multus\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysconfig\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-host\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874985    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-cni-multus\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.874995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-serviceca\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875036    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysconfig\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-kubelet\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-systemd\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7ee79e-263e-4f31-a115-9374f163a704-tmp-dir\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h44dt\" (UniqueName: \"kubernetes.io/projected/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-kube-api-access-h44dt\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875146    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-kubelet\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875153    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-env-overrides\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:14.930875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875198    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-systemd\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-systemd-units\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-systemd\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-slash\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-run-netns\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-log-socket\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-serviceca\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsjmn\" (UniqueName: \"kubernetes.io/projected/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-kube-api-access-zsjmn\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875439    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovnkube-config\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-cni-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-host\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875513    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-multus-certs\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-modprobe-d\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875547    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-slash\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7ee79e-263e-4f31-a115-9374f163a704-tmp-dir\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-system-cni-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-systemd-units\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.107328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875603    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-socket-dir-parent\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-etc-kubernetes\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875628    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-run-netns\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlpk\" (UniqueName: \"kubernetes.io/projected/6ef721b2-130c-4f92-824b-5b9039cee361-kube-api-access-7vlpk\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-var-lib-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-system-cni-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875688    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-modprobe-d\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysctl-d\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875661    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-log-socket\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-registration-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dxkv\" (UniqueName: \"kubernetes.io/projected/cb7ee79e-263e-4f31-a115-9374f163a704-kube-api-access-7dxkv\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-cni-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-registration-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875837    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-etc-kubernetes\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-cni-bin\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.875967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-conf-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:15.109097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876034    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-run-systemd\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-kubelet\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-multus-certs\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876073    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-var-lib-cni-bin\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-cni-netd\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-run\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.876150    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-var-lib-openvswitch\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876198    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-multus-conf-dir\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovn-node-metrics-cert\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:14.876211    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret podName:c8d1dd1d-8578-45a0-8f2a-cecb9128bb34 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:15.376194317 +0000 UTC m=+2.135721018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret") pod "global-pull-secret-syncer-nfh7p" (UID: "c8d1dd1d-8578-45a0-8f2a-cecb9128bb34") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876268    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-run\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-device-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-netns\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876317    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ef721b2-130c-4f92-824b-5b9039cee361-cni-binary-copy\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876329    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovnkube-script-lib\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.109659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-hostroot\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876360    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-sysctl-d\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876372    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-cni-netd\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876375    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ef721b2-130c-4f92-824b-5b9039cee361-multus-daemon-config\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-kubelet-config\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876488    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-kubelet-config\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-node-log\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp2r\" (UniqueName: \"kubernetes.io/projected/a5710820-7283-4fcf-b3a5-b0291c8f274f-kube-api-access-zfp2r\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876603    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-node-log\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-sys\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j54gs\" (UniqueName: \"kubernetes.io/projected/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-kube-api-access-j54gs\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-dbus\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-device-dir\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876776    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-tmp\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876797    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-dbus\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876830    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5710820-7283-4fcf-b3a5-b0291c8f274f-host-kubelet\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876879    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-sys\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.110879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876910    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-hostroot\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.876910    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ef721b2-130c-4f92-824b-5b9039cee361-host-run-netns\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.877231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6ef721b2-130c-4f92-824b-5b9039cee361-multus-daemon-config\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.877443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-etc-tuned\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.878423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5710820-7283-4fcf-b3a5-b0291c8f274f-ovn-node-metrics-cert\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.883300    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h44dt\" (UniqueName: \"kubernetes.io/projected/90f6fad6-42f0-46ba-b45d-0cc2c95c5c73-kube-api-access-h44dt\") pod \"tuned-nbqv4\" (UID: \"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73\") " pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.884533    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlpk\" (UniqueName: \"kubernetes.io/projected/6ef721b2-130c-4f92-824b-5b9039cee361-kube-api-access-7vlpk\") pod \"multus-brnls\" (UID: \"6ef721b2-130c-4f92-824b-5b9039cee361\") " pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.885099    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dxkv\" (UniqueName: \"kubernetes.io/projected/cb7ee79e-263e-4f31-a115-9374f163a704-kube-api-access-7dxkv\") pod \"node-resolver-mg6xr\" (UID: \"cb7ee79e-263e-4f31-a115-9374f163a704\") " pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.885160    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsjmn\" (UniqueName: \"kubernetes.io/projected/0e39c5c8-72f7-46d6-86d6-19ae56bb2da1-kube-api-access-zsjmn\") pod \"aws-ebs-csi-driver-node-8wpm5\" (UID: \"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.885942    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp2r\" (UniqueName: \"kubernetes.io/projected/a5710820-7283-4fcf-b3a5-b0291c8f274f-kube-api-access-zfp2r\") pod \"ovnkube-node-sxlfd\" (UID: \"a5710820-7283-4fcf-b3a5-b0291c8f274f\") " pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.886188    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j54gs\" (UniqueName: \"kubernetes.io/projected/6d88f24b-68e6-4d58-ac3e-98f1c68a55be-kube-api-access-j54gs\") pod \"node-ca-gc59t\" (UID: \"6d88f24b-68e6-4d58-ac3e-98f1c68a55be\") " pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:14.889002    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c23ff45096c6e80b4ac6b382d5677d.slice/crio-09c32da42cfc03c430b5303e2f3c10868340aae5ea8f03a7531409d7eda0a2b8 WatchSource:0}: Error finding container 09c32da42cfc03c430b5303e2f3c10868340aae5ea8f03a7531409d7eda0a2b8: Status 404 returned error can't find the container with id 09c32da42cfc03c430b5303e2f3c10868340aae5ea8f03a7531409d7eda0a2b8
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:14.889341    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c28a8c9ff7049a9f9a806acf5cb4eb.slice/crio-fcbd5b21e55f0d65c6746f0d1aeb42f6176c778adaa711a9e4e55c4a7ff18c1c WatchSource:0}: Error finding container fcbd5b21e55f0d65c6746f0d1aeb42f6176c778adaa711a9e4e55c4a7ff18c1c: Status 404 returned error can't find the container with id fcbd5b21e55f0d65c6746f0d1aeb42f6176c778adaa711a9e4e55c4a7ff18c1c
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.898318    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:14.991576    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.016868    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bf587"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.027018    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0058fb_09cb_4671_b2a3_1143bb567f59.slice/crio-bfb3e8d3b556789643332e4fc198c1c4900379bb351607dc5cbd404b7d16b50a WatchSource:0}: Error finding container bfb3e8d3b556789643332e4fc198c1c4900379bb351607dc5cbd404b7d16b50a: Status 404 returned error can't find the container with id bfb3e8d3b556789643332e4fc198c1c4900379bb351607dc5cbd404b7d16b50a
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.032937    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.042723    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7hs59"
Jul 01 19:21:15.111729 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.042897    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9c3578_a195_4a37_9ff9_a3ce0ea5c5a6.slice/crio-e532077e344640b3bbd57c5a20afddac1d674e23b0ad8758ab07d3503639bc78 WatchSource:0}: Error finding container e532077e344640b3bbd57c5a20afddac1d674e23b0ad8758ab07d3503639bc78: Status 404 returned error can't find the container with id e532077e344640b3bbd57c5a20afddac1d674e23b0ad8758ab07d3503639bc78
Jul 01 19:21:15.112929 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.049811    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f2a6f4_c9eb_4462_b413_3fef82d07445.slice/crio-4b529298b70d85aa2793564395f7051e8a167661da54b239f62e9e5adbd5286a WatchSource:0}: Error finding container 4b529298b70d85aa2793564395f7051e8a167661da54b239f62e9e5adbd5286a: Status 404 returned error can't find the container with id 4b529298b70d85aa2793564395f7051e8a167661da54b239f62e9e5adbd5286a
Jul 01 19:21:15.112929 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.066798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:15.112929 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.073983    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nbqv4"
Jul 01 19:21:15.112929 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.074373    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5710820_7283_4fcf_b3a5_b0291c8f274f.slice/crio-9661c877768ac3972ca2dc21ba912436104bf3670438445ad47446201fdd2fcc WatchSource:0}: Error finding container 9661c877768ac3972ca2dc21ba912436104bf3670438445ad47446201fdd2fcc: Status 404 returned error can't find the container with id 9661c877768ac3972ca2dc21ba912436104bf3670438445ad47446201fdd2fcc
Jul 01 19:21:15.112929 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.080983    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f6fad6_42f0_46ba_b45d_0cc2c95c5c73.slice/crio-6624a8785920e09c62e1a6f0635dc51a0fe08dfcc45e0672a76295cab08e2ca3 WatchSource:0}: Error finding container 6624a8785920e09c62e1a6f0635dc51a0fe08dfcc45e0672a76295cab08e2ca3: Status 404 returned error can't find the container with id 6624a8785920e09c62e1a6f0635dc51a0fe08dfcc45e0672a76295cab08e2ca3
Jul 01 19:21:15.112929 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.104359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gc59t"
Jul 01 19:21:15.114786 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.114766    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d88f24b_68e6_4d58_ac3e_98f1c68a55be.slice/crio-9b5c1f94224061fb9b20baf28b07e1843a3ea3f09fa6c1c7567928b0689ee792 WatchSource:0}: Error finding container 9b5c1f94224061fb9b20baf28b07e1843a3ea3f09fa6c1c7567928b0689ee792: Status 404 returned error can't find the container with id 9b5c1f94224061fb9b20baf28b07e1843a3ea3f09fa6c1c7567928b0689ee792
Jul 01 19:21:15.123542 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.123527    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-brnls"
Jul 01 19:21:15.131375 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.131357    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef721b2_130c_4f92_824b_5b9039cee361.slice/crio-5881e5697aa800b436a14e8d1f951225c6118687ae47a3d7eeea20ea80d52f1b WatchSource:0}: Error finding container 5881e5697aa800b436a14e8d1f951225c6118687ae47a3d7eeea20ea80d52f1b: Status 404 returned error can't find the container with id 5881e5697aa800b436a14e8d1f951225c6118687ae47a3d7eeea20ea80d52f1b
Jul 01 19:21:15.152691 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.152676    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5"
Jul 01 19:21:15.157198 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.157184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mg6xr"
Jul 01 19:21:15.162954 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.162936    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e39c5c8_72f7_46d6_86d6_19ae56bb2da1.slice/crio-2b522ea8dc18f426f7e541ddcea216e2a650ee2f320668bdbfdbb6fbb082bdc3 WatchSource:0}: Error finding container 2b522ea8dc18f426f7e541ddcea216e2a650ee2f320668bdbfdbb6fbb082bdc3: Status 404 returned error can't find the container with id 2b522ea8dc18f426f7e541ddcea216e2a650ee2f320668bdbfdbb6fbb082bdc3
Jul 01 19:21:15.171023 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:15.171006    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7ee79e_263e_4f31_a115_9374f163a704.slice/crio-be1d7efcdb8c1ca4ff3eb6a781880f02c7a2ee7b35dfd142638c5a1d429139f1 WatchSource:0}: Error finding container be1d7efcdb8c1ca4ff3eb6a781880f02c7a2ee7b35dfd142638c5a1d429139f1: Status 404 returned error can't find the container with id be1d7efcdb8c1ca4ff3eb6a781880f02c7a2ee7b35dfd142638c5a1d429139f1
Jul 01 19:21:15.191894 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.189911    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:15.278296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.278275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:15.278377 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.278356    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:15.278422 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.278396    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:16.278384341 +0000 UTC m=+3.037911044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:15.378870 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.378805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:15.379019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.378904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:15.379019 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.378980    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:15.379019 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.379002    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:15.379019 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.379006    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:15.379019 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.379015    2576 projected.go:196] Error preparing data for projected volume kube-api-access-b96vl for pod openshift-network-diagnostics/network-check-target-q4qsh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:15.379266 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.379069    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl podName:1e5dcab4-31fb-4c7f-a51e-5dc3a0792059 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:16.379049281 +0000 UTC m=+3.138575995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b96vl" (UniqueName: "kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl") pod "network-check-target-q4qsh" (UID: "1e5dcab4-31fb-4c7f-a51e-5dc3a0792059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:15.379266 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.379089    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret podName:c8d1dd1d-8578-45a0-8f2a-cecb9128bb34 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:16.379079489 +0000 UTC m=+3.138606195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret") pod "global-pull-secret-syncer-nfh7p" (UID: "c8d1dd1d-8578-45a0-8f2a-cecb9128bb34") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:15.464137 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.464103    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 19:21:15.719259 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.719154    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:16:14 +0000 UTC" deadline="2028-03-15 05:25:09.525988198 +0000 UTC"
Jul 01 19:21:15.719259 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.719218    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14938h3m53.806774616s"
Jul 01 19:21:15.759598 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.759572    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:15.759732 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.759704    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:15.760114 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.760093    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:15.760216 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:15.760194    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:15.767824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.767778    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal" event={"ID":"c1c28a8c9ff7049a9f9a806acf5cb4eb","Type":"ContainerStarted","Data":"fcbd5b21e55f0d65c6746f0d1aeb42f6176c778adaa711a9e4e55c4a7ff18c1c"}
Jul 01 19:21:15.772401 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.772242    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-brnls" event={"ID":"6ef721b2-130c-4f92-824b-5b9039cee361","Type":"ContainerStarted","Data":"5881e5697aa800b436a14e8d1f951225c6118687ae47a3d7eeea20ea80d52f1b"}
Jul 01 19:21:15.782077 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.781860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nbqv4" event={"ID":"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73","Type":"ContainerStarted","Data":"6624a8785920e09c62e1a6f0635dc51a0fe08dfcc45e0672a76295cab08e2ca3"}
Jul 01 19:21:15.795366 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.795263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"9661c877768ac3972ca2dc21ba912436104bf3670438445ad47446201fdd2fcc"}
Jul 01 19:21:15.818735 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.818707    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerStarted","Data":"4b529298b70d85aa2793564395f7051e8a167661da54b239f62e9e5adbd5286a"}
Jul 01 19:21:15.844939 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.844876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vxzzn" event={"ID":"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6","Type":"ContainerStarted","Data":"e532077e344640b3bbd57c5a20afddac1d674e23b0ad8758ab07d3503639bc78"}
Jul 01 19:21:15.858138 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.858073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bf587" event={"ID":"af0058fb-09cb-4671-b2a3-1143bb567f59","Type":"ContainerStarted","Data":"bfb3e8d3b556789643332e4fc198c1c4900379bb351607dc5cbd404b7d16b50a"}
Jul 01 19:21:15.867842 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.867816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal" event={"ID":"24c23ff45096c6e80b4ac6b382d5677d","Type":"ContainerStarted","Data":"09c32da42cfc03c430b5303e2f3c10868340aae5ea8f03a7531409d7eda0a2b8"}
Jul 01 19:21:15.888123 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.888073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mg6xr" event={"ID":"cb7ee79e-263e-4f31-a115-9374f163a704","Type":"ContainerStarted","Data":"be1d7efcdb8c1ca4ff3eb6a781880f02c7a2ee7b35dfd142638c5a1d429139f1"}
Jul 01 19:21:15.899742 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.899716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5" event={"ID":"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1","Type":"ContainerStarted","Data":"2b522ea8dc18f426f7e541ddcea216e2a650ee2f320668bdbfdbb6fbb082bdc3"}
Jul 01 19:21:15.912715 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:15.912687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gc59t" event={"ID":"6d88f24b-68e6-4d58-ac3e-98f1c68a55be","Type":"ContainerStarted","Data":"9b5c1f94224061fb9b20baf28b07e1843a3ea3f09fa6c1c7567928b0689ee792"}
Jul 01 19:21:16.289415 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:16.288829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:16.289415 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.289019    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:16.289415 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.289076    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:18.28905819 +0000 UTC m=+5.048584893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:16.389326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:16.389436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.389580    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.389636    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret podName:c8d1dd1d-8578-45a0-8f2a-cecb9128bb34 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:18.389619092 +0000 UTC m=+5.149145794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret") pod "global-pull-secret-syncer-nfh7p" (UID: "c8d1dd1d-8578-45a0-8f2a-cecb9128bb34") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.390001    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.390020    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.390031    2576 projected.go:196] Error preparing data for projected volume kube-api-access-b96vl for pod openshift-network-diagnostics/network-check-target-q4qsh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:16.390188 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.390073    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl podName:1e5dcab4-31fb-4c7f-a51e-5dc3a0792059 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:18.390058745 +0000 UTC m=+5.149585446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b96vl" (UniqueName: "kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl") pod "network-check-target-q4qsh" (UID: "1e5dcab4-31fb-4c7f-a51e-5dc3a0792059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:16.720062 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:16.719975    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 19:16:14 +0000 UTC" deadline="2028-04-17 15:27:29.812019172 +0000 UTC"
Jul 01 19:21:16.720062 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:16.720014    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15740h6m13.092010161s"
Jul 01 19:21:16.744676 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:16.744649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:16.744824 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:16.744770    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:17.745048 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:17.745018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:17.745526 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:17.745162    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:17.745597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:17.745544    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:17.745667 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:17.745645    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:18.306704 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:18.306667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:18.306906 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.306884    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:18.306976 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.306953    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:22.306931474 +0000 UTC m=+9.066458190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:18.407688 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:18.407652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:18.407846 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:18.407706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:18.407923 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.407845    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:18.407923 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.407863    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:18.407923 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.407877    2576 projected.go:196] Error preparing data for projected volume kube-api-access-b96vl for pod openshift-network-diagnostics/network-check-target-q4qsh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:18.408078 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.407933    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl podName:1e5dcab4-31fb-4c7f-a51e-5dc3a0792059 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:22.407915853 +0000 UTC m=+9.167442553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b96vl" (UniqueName: "kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl") pod "network-check-target-q4qsh" (UID: "1e5dcab4-31fb-4c7f-a51e-5dc3a0792059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:18.408240 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.408156    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:18.408240 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.408210    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret podName:c8d1dd1d-8578-45a0-8f2a-cecb9128bb34 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:22.408195733 +0000 UTC m=+9.167722451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret") pod "global-pull-secret-syncer-nfh7p" (UID: "c8d1dd1d-8578-45a0-8f2a-cecb9128bb34") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:18.744778 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:18.744703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:18.744933 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:18.744838    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:19.745104 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:19.745067    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:19.745738 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:19.745202    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:19.745738 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:19.745613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:19.745738 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:19.745731    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:20.744667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:20.744631    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:20.744850 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:20.744762    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:21.748137 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:21.748103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:21.748649 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:21.748232    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:21.748649 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:21.748287    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:21.748649 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:21.748413    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:22.341622 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:22.340923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:22.341622 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.341103    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:22.341622 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.341294    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:30.341271622 +0000 UTC m=+17.100798336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:22.442578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:22.442496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:22.442578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:22.442540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:22.442783 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.442648    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:22.442783 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.442715    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret podName:c8d1dd1d-8578-45a0-8f2a-cecb9128bb34 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:30.44269469 +0000 UTC m=+17.202221409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret") pod "global-pull-secret-syncer-nfh7p" (UID: "c8d1dd1d-8578-45a0-8f2a-cecb9128bb34") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:22.442783 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.442741    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:22.442783 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.442763    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:22.442783 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.442776    2576 projected.go:196] Error preparing data for projected volume kube-api-access-b96vl for pod openshift-network-diagnostics/network-check-target-q4qsh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:22.443047 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.442822    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl podName:1e5dcab4-31fb-4c7f-a51e-5dc3a0792059 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:30.442810248 +0000 UTC m=+17.202336961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b96vl" (UniqueName: "kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl") pod "network-check-target-q4qsh" (UID: "1e5dcab4-31fb-4c7f-a51e-5dc3a0792059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:22.745245 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:22.745121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:22.745391 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:22.745270    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:23.748247 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:23.747986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:23.748247 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:23.747987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:23.748247 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:23.748119    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:23.748247 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:23.748180    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:24.745275 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:24.745200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:24.745423 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:24.745328    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:25.745382 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:25.745345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:25.745853 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:25.745506    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:25.745853 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:25.745693    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:25.745853 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:25.745825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:26.745339 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:26.745307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:26.745487 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:26.745434    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:27.744853 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:27.744819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:27.745097 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:27.744960    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:27.745097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:27.744998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:27.745238 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:27.745116    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:28.744978 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:28.744940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:28.745477 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:28.745066    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:29.744911 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:29.744879    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:29.745083 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:29.744923    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:29.745083 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:29.745004    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:29.745512 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:29.745148    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:30.403180 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:30.403159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:30.403282 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.403256    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:30.403320 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.403311    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:46.403296102 +0000 UTC m=+33.162822802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:30.503782 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:30.503719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:30.503782 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:30.503757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:30.503943 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.503866    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:30.503943 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.503881    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:30.503943 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.503890    2576 projected.go:196] Error preparing data for projected volume kube-api-access-b96vl for pod openshift-network-diagnostics/network-check-target-q4qsh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:30.503943 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.503928    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl podName:1e5dcab4-31fb-4c7f-a51e-5dc3a0792059 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:46.50391753 +0000 UTC m=+33.263444242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b96vl" (UniqueName: "kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl") pod "network-check-target-q4qsh" (UID: "1e5dcab4-31fb-4c7f-a51e-5dc3a0792059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:30.503943 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.503864    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:30.504104 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.503995    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret podName:c8d1dd1d-8578-45a0-8f2a-cecb9128bb34 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:46.503980603 +0000 UTC m=+33.263507302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret") pod "global-pull-secret-syncer-nfh7p" (UID: "c8d1dd1d-8578-45a0-8f2a-cecb9128bb34") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:30.745377 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:30.745342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:30.745776 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:30.745490    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:31.744788 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:31.744755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:31.745030 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:31.744914    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:31.745030 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:31.744953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:31.745125 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:31.745042    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:32.745183 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:32.745155    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:32.745595 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:32.745264    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:33.745305 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:33.745271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:33.745786 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:33.745374    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:33.745786 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:33.745404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:33.745786 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:33.745576    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:34.744564 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:34.744531    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:34.744683 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:34.744641    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:34.958056 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:34.958025    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal" event={"ID":"24c23ff45096c6e80b4ac6b382d5677d","Type":"ContainerStarted","Data":"94e096ae1f91d9b105b859623188269ab4f34561e41d8c1205f4bc459f80c2cd"}
Jul 01 19:21:34.976615 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:34.976257    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-234.ec2.internal" podStartSLOduration=20.976238487 podStartE2EDuration="20.976238487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:34.975969704 +0000 UTC m=+21.735496426" watchObservedRunningTime="2026-07-01 19:21:34.976238487 +0000 UTC m=+21.735765209"
Jul 01 19:21:35.745616 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.745315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:35.745726 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.745381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:35.745787 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:35.745721    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:35.745844 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:35.745775    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:35.961403 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.961330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bf587" event={"ID":"af0058fb-09cb-4671-b2a3-1143bb567f59","Type":"ContainerStarted","Data":"665e8efb8748549d63079a72629c011c9bc773b980524668559f64c2abdde28e"}
Jul 01 19:21:35.962777 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.962750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mg6xr" event={"ID":"cb7ee79e-263e-4f31-a115-9374f163a704","Type":"ContainerStarted","Data":"aa74b085568903b5759b04f8843b6599567a10a749992cd573088cc4b8a8aaf4"}
Jul 01 19:21:35.964040 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.964014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5" event={"ID":"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1","Type":"ContainerStarted","Data":"7c074a08c1e1ad9b63d69542a9143d81f21dd1e58ba3ec36d337be605dfcbe1e"}
Jul 01 19:21:35.965263 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.965231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gc59t" event={"ID":"6d88f24b-68e6-4d58-ac3e-98f1c68a55be","Type":"ContainerStarted","Data":"2090768a9f1a326843399789f187285286fa2ffe9b235e1dfa54f6d8722736c4"}
Jul 01 19:21:35.966489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.966466    2576 generic.go:356] "Generic (PLEG): container finished" podID="c1c28a8c9ff7049a9f9a806acf5cb4eb" containerID="c52c6ef91f601022d7ce6daeae62804220d66a07862bc624e27bdf30875cb9b9" exitCode=0
Jul 01 19:21:35.966581 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.966540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal" event={"ID":"c1c28a8c9ff7049a9f9a806acf5cb4eb","Type":"ContainerDied","Data":"c52c6ef91f601022d7ce6daeae62804220d66a07862bc624e27bdf30875cb9b9"}
Jul 01 19:21:35.967874 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.967855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-brnls" event={"ID":"6ef721b2-130c-4f92-824b-5b9039cee361","Type":"ContainerStarted","Data":"994a03d39c3b6623b7e587712b6b76124efa5917a4c582bbcd8be7ea1557c913"}
Jul 01 19:21:35.969276 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.969259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nbqv4" event={"ID":"90f6fad6-42f0-46ba-b45d-0cc2c95c5c73","Type":"ContainerStarted","Data":"53d58063bb93a0d54f5f08c908219c11a2e0f86a8fb5e3b18f61162f23e8c08b"}
Jul 01 19:21:35.971850 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.971830    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"75f0581d91a0f2b26b5e3ea21d0509e3782e8981c2a29162736dc0bedf896f95"}
Jul 01 19:21:35.971935 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.971889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"1bee0fe4fd60bbcbd84b2a299b6b104f42aca452c6d6db9867f65bbabf496e59"}
Jul 01 19:21:35.971935 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.971927    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"428f274ed989209c5c31c2f0b395526550995f7f2fb517bc53245d8e12916264"}
Jul 01 19:21:35.972038 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.971965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"78b6ee133b7f8fd30ef2a5be02e46b0d8a96316e8b176e63e86bc581a93d56bf"}
Jul 01 19:21:35.972038 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.972013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"080783b2e4977ece7b6561f67a98f04801fb3742d6c8ff36252a426601eb00c1"}
Jul 01 19:21:35.972142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.972050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"464abad44d2a2f99628e5bec38fbf20b2d6258e1b4560ff8db890ee37fa08ea5"}
Jul 01 19:21:35.973267 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.973250    2576 generic.go:356] "Generic (PLEG): container finished" podID="14f2a6f4-c9eb-4462-b413-3fef82d07445" containerID="9ffcf924b716d9a4f0a6d1be1bb9fb75ec438a076c001e1002ce29b879df43f0" exitCode=0
Jul 01 19:21:35.973366 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.973308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerDied","Data":"9ffcf924b716d9a4f0a6d1be1bb9fb75ec438a076c001e1002ce29b879df43f0"}
Jul 01 19:21:35.974808 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.974789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vxzzn" event={"ID":"ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6","Type":"ContainerStarted","Data":"7cf7b566e18e1fe3502ecd1bffe934bd1242c31db4aa2ef23863fb53d3028c1b"}
Jul 01 19:21:35.977147 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.977111    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bf587" podStartSLOduration=3.340406574 podStartE2EDuration="22.977100748s" totalImagesPullingTime="19.636694174s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:35.976959311 +0000 UTC m=+22.736486032" watchObservedRunningTime="2026-07-01 19:21:35.977100748 +0000 UTC m=+22.736627468"
Jul 01 19:21:35.995500 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:35.995468    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mg6xr" podStartSLOduration=2.492018021 podStartE2EDuration="21.995443969s" totalImagesPullingTime="19.503425948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:35.994958037 +0000 UTC m=+22.754484758" watchObservedRunningTime="2026-07-01 19:21:35.995443969 +0000 UTC m=+22.754970690"
Jul 01 19:21:36.040019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.039979    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gc59t" podStartSLOduration=2.470446232 podStartE2EDuration="22.039967822s" totalImagesPullingTime="19.56952159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:36.039726249 +0000 UTC m=+22.799252971" watchObservedRunningTime="2026-07-01 19:21:36.039967822 +0000 UTC m=+22.799494543"
Jul 01 19:21:36.064677 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.064644    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-brnls" podStartSLOduration=2.197697171 podStartE2EDuration="22.064631095s" totalImagesPullingTime="19.866933924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:36.064244906 +0000 UTC m=+22.823771788" watchObservedRunningTime="2026-07-01 19:21:36.064631095 +0000 UTC m=+22.824157813"
Jul 01 19:21:36.117594 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.117550    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nbqv4" podStartSLOduration=2.512560642 podStartE2EDuration="22.117537482s" totalImagesPullingTime="19.60497684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:36.117041932 +0000 UTC m=+22.876568652" watchObservedRunningTime="2026-07-01 19:21:36.117537482 +0000 UTC m=+22.877064197"
Jul 01 19:21:36.143268 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.143231    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vxzzn" podStartSLOduration=3.522573269 podStartE2EDuration="23.143218571s" totalImagesPullingTime="19.620645302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:36.143119504 +0000 UTC m=+22.902646224" watchObservedRunningTime="2026-07-01 19:21:36.143218571 +0000 UTC m=+22.902745291"
Jul 01 19:21:36.744380 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.744349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:36.744682 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:36.744531    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:36.902530 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.902503    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 19:21:36.978950 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.978916    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5" event={"ID":"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1","Type":"ContainerStarted","Data":"09d1219a68a0dbf51f39c061f8a275a3d7b23e94eb55161d631f958c6667ee55"}
Jul 01 19:21:36.981194 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.981139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal" event={"ID":"c1c28a8c9ff7049a9f9a806acf5cb4eb","Type":"ContainerStarted","Data":"3044605a5d9885af95621f292907d56e0944a53655a7449b880c1e05b9efc522"}
Jul 01 19:21:36.997211 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:36.997132    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-234.ec2.internal" podStartSLOduration=22.997113535 podStartE2EDuration="22.997113535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:36.996699078 +0000 UTC m=+23.756225799" watchObservedRunningTime="2026-07-01 19:21:36.997113535 +0000 UTC m=+23.756640281"
Jul 01 19:21:37.740563 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:37.740434    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T19:21:36.902526126Z","UUID":"e67c82af-7319-47de-a8ed-d89439fa689a","Handler":null,"Name":"","Endpoint":""}
Jul 01 19:21:37.742267 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:37.742245    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 19:21:37.742392 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:37.742284    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 19:21:37.744630 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:37.744606    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:37.744749 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:37.744722    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:37.744872 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:37.744606    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:37.745026 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:37.744966    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:38.744503 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:38.744477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:38.744942 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:38.744596    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:38.989915 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:38.989878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"0acba14d62ae430d1a206a41a07488de1403f43876b5c1ea2ce18cdbd99a6fb0"}
Jul 01 19:21:38.992161 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:38.992130    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5" event={"ID":"0e39c5c8-72f7-46d6-86d6-19ae56bb2da1","Type":"ContainerStarted","Data":"883af10b18d40f95e0830a48bd8f9b05884848c37d35af3e028f2d5e6adbfdf2"}
Jul 01 19:21:39.011917 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:39.011842    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wpm5" podStartSLOduration=7.15472359 podStartE2EDuration="25.01182786s" totalImagesPullingTime="17.85710427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:39.011748762 +0000 UTC m=+25.771275484" watchObservedRunningTime="2026-07-01 19:21:39.01182786 +0000 UTC m=+25.771354621"
Jul 01 19:21:39.612063 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:39.611821    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:39.612532 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:39.612508    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:39.744434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:39.744404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:39.744586 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:39.744537    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:39.744913 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:39.744605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:39.744913 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:39.744714    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:39.994637 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:39.994563    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:39.994937 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:39.994920    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vxzzn"
Jul 01 19:21:40.744371 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:40.744342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:40.744583 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:40.744445    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:41.000478 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.000132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" event={"ID":"a5710820-7283-4fcf-b3a5-b0291c8f274f","Type":"ContainerStarted","Data":"0a0014772f64d730283624ae7a4afe61e2f14855e1c669996f6fcf2045ac44fa"}
Jul 01 19:21:41.000478 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.000438    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:41.001332 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.000503    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:41.001332 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.000545    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:41.002215 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.002187    2576 generic.go:356] "Generic (PLEG): container finished" podID="14f2a6f4-c9eb-4462-b413-3fef82d07445" containerID="3f52f6f58168b59a091d8e6f7a17e51121238110b6502a268ca037127eb2209c" exitCode=0
Jul 01 19:21:41.002322 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.002218    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerDied","Data":"3f52f6f58168b59a091d8e6f7a17e51121238110b6502a268ca037127eb2209c"}
Jul 01 19:21:41.015936 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.015916    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:41.016047 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.016006    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:21:41.055906 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.055865    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd" podStartSLOduration=7.165549019 podStartE2EDuration="27.055852693s" totalImagesPullingTime="19.890303674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:41.035577385 +0000 UTC m=+27.795104106" watchObservedRunningTime="2026-07-01 19:21:41.055852693 +0000 UTC m=+27.815379467"
Jul 01 19:21:41.744568 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.744539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:41.744757 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:41.744651    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:41.744757 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:41.744714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:41.744859 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:41.744836    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:42.588386 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:42.588357    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nfh7p"]
Jul 01 19:21:42.588744 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:42.588463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:42.588744 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:42.588568    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:42.593486 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:42.593445    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4qsh"]
Jul 01 19:21:42.593600 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:42.593564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:42.593675 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:42.593654    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:42.594889 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:42.594865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r2hbh"]
Jul 01 19:21:42.594988 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:42.594958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:42.595096 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:42.595061    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:43.008081 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:43.008046    2576 generic.go:356] "Generic (PLEG): container finished" podID="14f2a6f4-c9eb-4462-b413-3fef82d07445" containerID="91ccd50055fd59791151d98418eed0f94c4020974972d76f11891d0d9e67814b" exitCode=0
Jul 01 19:21:43.008235 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:43.008124    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerDied","Data":"91ccd50055fd59791151d98418eed0f94c4020974972d76f11891d0d9e67814b"}
Jul 01 19:21:43.745651 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:43.745621    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:43.746109 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:43.745729    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:44.744402 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:44.744327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:44.744402 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:44.744351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:44.744577 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:44.744443    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:44.744577 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:44.744557    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:45.014296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:45.014228    2576 generic.go:356] "Generic (PLEG): container finished" podID="14f2a6f4-c9eb-4462-b413-3fef82d07445" containerID="b3654cefdc43126bc361f560ca2ef7fd189a60c5c7c1b850800f04c68a4d216b" exitCode=0
Jul 01 19:21:45.014296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:45.014269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerDied","Data":"b3654cefdc43126bc361f560ca2ef7fd189a60c5c7c1b850800f04c68a4d216b"}
Jul 01 19:21:45.745019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:45.744993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:45.745126 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:45.745088    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:46.426826 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:46.426599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:46.427225 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.426709    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:46.427225 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.426943    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:18.426924422 +0000 UTC m=+65.186451122 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 19:21:46.527756 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:46.527723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:46.527933 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:46.527808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:46.527933 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.527895    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:46.527933 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.527912    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 19:21:46.528078 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.527936    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 19:21:46.528078 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.527947    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret podName:c8d1dd1d-8578-45a0-8f2a-cecb9128bb34 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:18.527931183 +0000 UTC m=+65.287457884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret") pod "global-pull-secret-syncer-nfh7p" (UID: "c8d1dd1d-8578-45a0-8f2a-cecb9128bb34") : object "kube-system"/"original-pull-secret" not registered
Jul 01 19:21:46.528078 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.527949    2576 projected.go:196] Error preparing data for projected volume kube-api-access-b96vl for pod openshift-network-diagnostics/network-check-target-q4qsh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:46.528078 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.527987    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl podName:1e5dcab4-31fb-4c7f-a51e-5dc3a0792059 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:18.527978279 +0000 UTC m=+65.287504991 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b96vl" (UniqueName: "kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl") pod "network-check-target-q4qsh" (UID: "1e5dcab4-31fb-4c7f-a51e-5dc3a0792059") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 19:21:46.745246 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:46.745175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:46.745246 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:46.745202    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:46.745411 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.745289    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r2hbh" podUID="a22a2075-444a-40ee-8406-7cd45cd55403"
Jul 01 19:21:46.745411 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:46.745394    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nfh7p" podUID="c8d1dd1d-8578-45a0-8f2a-cecb9128bb34"
Jul 01 19:21:47.744807 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:47.744780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:47.745212 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:47.744880    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q4qsh" podUID="1e5dcab4-31fb-4c7f-a51e-5dc3a0792059"
Jul 01 19:21:47.991214 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:47.991183    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-234.ec2.internal" event="NodeReady"
Jul 01 19:21:47.991375 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:47.991305    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 19:21:48.025931 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.025901    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69fb5944f6-lff5w"]
Jul 01 19:21:48.031156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.031134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.034298 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.034275    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 19:21:48.034426 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.034354    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 19:21:48.034620 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.034599    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 19:21:48.034752 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.034733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wklt5\""
Jul 01 19:21:48.043216 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.043192    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 19:21:48.047290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.047268    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ch6qk"]
Jul 01 19:21:48.050839 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.050821    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69fb5944f6-lff5w"]
Jul 01 19:21:48.050929 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.050856    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2mnld"]
Jul 01 19:21:48.050988 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.050949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.053603 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.053554    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 19:21:48.053695 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.053607    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8prn\""
Jul 01 19:21:48.053895 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.053876    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 19:21:48.054332 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.054316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.057051 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.057011    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 19:21:48.057153 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.057120    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 19:21:48.057247 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.057151    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 19:21:48.057247 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.057169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d8sfh\""
Jul 01 19:21:48.060330 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.060313    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mnld"]
Jul 01 19:21:48.062994 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.062960    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ch6qk"]
Jul 01 19:21:48.138849 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.138821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-installation-pull-secrets\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.138849 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.138853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlccw\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-kube-api-access-jlccw\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.139035 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.138877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-bound-sa-token\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.139035 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.138908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sfr\" (UniqueName: \"kubernetes.io/projected/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-kube-api-access-g6sfr\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.139035 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-trusted-ca\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.139156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139032    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-config-volume\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.139156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62e6f8b2-4af9-4576-b751-aa4176320cbd-ca-trust-extracted\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.139229 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.139264 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-certificates\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.139305 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.139305 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-tmp-dir\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.139378 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.139324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-image-registry-private-configuration\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.239784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.239915 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-certificates\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.239915 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.239915 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-tmp-dir\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.239915 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-image-registry-private-configuration\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.239915 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5ch\" (UniqueName: \"kubernetes.io/projected/b55a5e16-7871-451f-a5c6-be4314ca7fb5-kube-api-access-nf5ch\") pod \"ingress-canary-2mnld\" (UID: \"b55a5e16-7871-451f-a5c6-be4314ca7fb5\") " pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55a5e16-7871-451f-a5c6-be4314ca7fb5-cert\") pod \"ingress-canary-2mnld\" (UID: \"b55a5e16-7871-451f-a5c6-be4314ca7fb5\") " pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.239934    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.239955    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fb5944f6-lff5w: secret "image-registry-tls" not found
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-installation-pull-secrets\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.239963    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.239982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlccw\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-kube-api-access-jlccw\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240011    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-bound-sa-token\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.240027    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls podName:62e6f8b2-4af9-4576-b751-aa4176320cbd nodeName:}" failed. No retries permitted until 2026-07-01 19:21:48.740005811 +0000 UTC m=+35.499532513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls") pod "image-registry-69fb5944f6-lff5w" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd") : secret "image-registry-tls" not found
Jul 01 19:21:48.240184 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.240165    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls podName:0d95f075-ee0e-498d-8cb0-52a0d3080e5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:48.740133962 +0000 UTC m=+35.499660667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls") pod "dns-default-ch6qk" (UID: "0d95f075-ee0e-498d-8cb0-52a0d3080e5c") : secret "dns-default-metrics-tls" not found
Jul 01 19:21:48.240637 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sfr\" (UniqueName: \"kubernetes.io/projected/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-kube-api-access-g6sfr\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.240637 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-trusted-ca\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.240637 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-config-volume\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.240637 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62e6f8b2-4af9-4576-b751-aa4176320cbd-ca-trust-extracted\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.240637 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240426    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-tmp-dir\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.240637 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240614    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-certificates\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.240897 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62e6f8b2-4af9-4576-b751-aa4176320cbd-ca-trust-extracted\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.240991 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.240966    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-config-volume\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.241302 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.241249    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-trusted-ca\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.244887 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.244825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-image-registry-private-configuration\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.244887 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.244834    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-installation-pull-secrets\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.249436 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.249417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-bound-sa-token\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.250153 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.250105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlccw\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-kube-api-access-jlccw\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.250153 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.250147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sfr\" (UniqueName: \"kubernetes.io/projected/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-kube-api-access-g6sfr\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.341482 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.341437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5ch\" (UniqueName: \"kubernetes.io/projected/b55a5e16-7871-451f-a5c6-be4314ca7fb5-kube-api-access-nf5ch\") pod \"ingress-canary-2mnld\" (UID: \"b55a5e16-7871-451f-a5c6-be4314ca7fb5\") " pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.341610 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.341502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55a5e16-7871-451f-a5c6-be4314ca7fb5-cert\") pod \"ingress-canary-2mnld\" (UID: \"b55a5e16-7871-451f-a5c6-be4314ca7fb5\") " pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.343832 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.343810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b55a5e16-7871-451f-a5c6-be4314ca7fb5-cert\") pod \"ingress-canary-2mnld\" (UID: \"b55a5e16-7871-451f-a5c6-be4314ca7fb5\") " pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.350668 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.350648    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5ch\" (UniqueName: \"kubernetes.io/projected/b55a5e16-7871-451f-a5c6-be4314ca7fb5-kube-api-access-nf5ch\") pod \"ingress-canary-2mnld\" (UID: \"b55a5e16-7871-451f-a5c6-be4314ca7fb5\") " pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.369669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.369650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mnld"
Jul 01 19:21:48.745405 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.745372    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.745413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.745475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.745595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.745622    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.745685    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.745697    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fb5944f6-lff5w: secret "image-registry-tls" not found
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.745698    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls podName:0d95f075-ee0e-498d-8cb0-52a0d3080e5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:49.745675474 +0000 UTC m=+36.505202177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls") pod "dns-default-ch6qk" (UID: "0d95f075-ee0e-498d-8cb0-52a0d3080e5c") : secret "dns-default-metrics-tls" not found
Jul 01 19:21:48.746085 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:48.745760    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls podName:62e6f8b2-4af9-4576-b751-aa4176320cbd nodeName:}" failed. No retries permitted until 2026-07-01 19:21:49.745745259 +0000 UTC m=+36.505271974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls") pod "image-registry-69fb5944f6-lff5w" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd") : secret "image-registry-tls" not found
Jul 01 19:21:48.748611 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.748566    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:21:48.748731 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.748634    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:21:48.748731 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:48.748652    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzk4c\""
Jul 01 19:21:49.543245 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:49.543128    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mnld"]
Jul 01 19:21:49.546286 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:49.546265    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55a5e16_7871_451f_a5c6_be4314ca7fb5.slice/crio-c163b949d6ef9da7cd807fa6fe1f96e26eb4b98cbb784d28abac011fe916563a WatchSource:0}: Error finding container c163b949d6ef9da7cd807fa6fe1f96e26eb4b98cbb784d28abac011fe916563a: Status 404 returned error can't find the container with id c163b949d6ef9da7cd807fa6fe1f96e26eb4b98cbb784d28abac011fe916563a
Jul 01 19:21:49.745228 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:49.745200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:21:49.749174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:49.749152    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 19:21:49.749174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:49.749156    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbvbl\""
Jul 01 19:21:49.749782 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:49.749552    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 19:21:49.752890 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:49.752866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:49.753008 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:49.752991    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:21:49.753073 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:49.753012    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fb5944f6-lff5w: secret "image-registry-tls" not found
Jul 01 19:21:49.753073 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:49.753001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:49.753174 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:49.753115    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls podName:62e6f8b2-4af9-4576-b751-aa4176320cbd nodeName:}" failed. No retries permitted until 2026-07-01 19:21:51.753094825 +0000 UTC m=+38.512621524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls") pod "image-registry-69fb5944f6-lff5w" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd") : secret "image-registry-tls" not found
Jul 01 19:21:49.753174 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:49.753163    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:21:49.753277 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:49.753231    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls podName:0d95f075-ee0e-498d-8cb0-52a0d3080e5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:51.753213682 +0000 UTC m=+38.512740382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls") pod "dns-default-ch6qk" (UID: "0d95f075-ee0e-498d-8cb0-52a0d3080e5c") : secret "dns-default-metrics-tls" not found
Jul 01 19:21:50.026660 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:50.026604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mnld" event={"ID":"b55a5e16-7871-451f-a5c6-be4314ca7fb5","Type":"ContainerStarted","Data":"c163b949d6ef9da7cd807fa6fe1f96e26eb4b98cbb784d28abac011fe916563a"}
Jul 01 19:21:50.029044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:50.029023    2576 generic.go:356] "Generic (PLEG): container finished" podID="14f2a6f4-c9eb-4462-b413-3fef82d07445" containerID="28c875cf17d2376ba91f3be6cb353c4f45521ca18b506ac3634772ad67cd9fbf" exitCode=0
Jul 01 19:21:50.029126 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:50.029054    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerDied","Data":"28c875cf17d2376ba91f3be6cb353c4f45521ca18b506ac3634772ad67cd9fbf"}
Jul 01 19:21:51.034656 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:51.034618    2576 generic.go:356] "Generic (PLEG): container finished" podID="14f2a6f4-c9eb-4462-b413-3fef82d07445" containerID="aa29c7ed36514d7a5553110a3a4e45a2ce3493734aa35b21e2f77741f0a22e87" exitCode=0
Jul 01 19:21:51.035015 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:51.034662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerDied","Data":"aa29c7ed36514d7a5553110a3a4e45a2ce3493734aa35b21e2f77741f0a22e87"}
Jul 01 19:21:51.768168 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:51.768143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:51.768168 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:51.768175    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:51.768394 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:51.768306    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:21:51.768394 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:51.768312    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:21:51.768394 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:51.768330    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fb5944f6-lff5w: secret "image-registry-tls" not found
Jul 01 19:21:51.768394 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:51.768354    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls podName:0d95f075-ee0e-498d-8cb0-52a0d3080e5c nodeName:}" failed. No retries permitted until 2026-07-01 19:21:55.768337089 +0000 UTC m=+42.527863805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls") pod "dns-default-ch6qk" (UID: "0d95f075-ee0e-498d-8cb0-52a0d3080e5c") : secret "dns-default-metrics-tls" not found
Jul 01 19:21:51.768394 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:51.768378    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls podName:62e6f8b2-4af9-4576-b751-aa4176320cbd nodeName:}" failed. No retries permitted until 2026-07-01 19:21:55.768363499 +0000 UTC m=+42.527890218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls") pod "image-registry-69fb5944f6-lff5w" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd") : secret "image-registry-tls" not found
Jul 01 19:21:52.040109 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:52.040080    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7hs59" event={"ID":"14f2a6f4-c9eb-4462-b413-3fef82d07445","Type":"ContainerStarted","Data":"37e5cf3deb0799b6bc496ffbf912107f05e6b538390ca77977717b1a2c75aa9c"}
Jul 01 19:21:52.066479 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:52.066423    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7hs59" podStartSLOduration=8.594367406 podStartE2EDuration="39.06640946s" totalImagesPullingTime="30.472042054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:52.064585529 +0000 UTC m=+38.824112250" watchObservedRunningTime="2026-07-01 19:21:52.06640946 +0000 UTC m=+38.825936180"
Jul 01 19:21:53.044211 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:53.044042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mnld" event={"ID":"b55a5e16-7871-451f-a5c6-be4314ca7fb5","Type":"ContainerStarted","Data":"b11063daff7585fb48149ccaaa471ed16d0b482462d4fc20d595946605e68613"}
Jul 01 19:21:53.064249 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:53.064203    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2mnld" podStartSLOduration=2.497548057 podStartE2EDuration="5.064188858s" totalImagesPullingTime="2.566640801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:21:53.063774423 +0000 UTC m=+39.823301144" watchObservedRunningTime="2026-07-01 19:21:53.064188858 +0000 UTC m=+39.823715579"
Jul 01 19:21:55.796232 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:55.796204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:21:55.796592 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:55.796239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:21:55.796592 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:55.796336    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:21:55.796592 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:55.796342    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:21:55.796592 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:55.796358    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fb5944f6-lff5w: secret "image-registry-tls" not found
Jul 01 19:21:55.796592 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:55.796380    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls podName:0d95f075-ee0e-498d-8cb0-52a0d3080e5c nodeName:}" failed. No retries permitted until 2026-07-01 19:22:03.79636826 +0000 UTC m=+50.555894959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls") pod "dns-default-ch6qk" (UID: "0d95f075-ee0e-498d-8cb0-52a0d3080e5c") : secret "dns-default-metrics-tls" not found
Jul 01 19:21:55.796592 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:55.796403    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls podName:62e6f8b2-4af9-4576-b751-aa4176320cbd nodeName:}" failed. No retries permitted until 2026-07-01 19:22:03.796389794 +0000 UTC m=+50.555916492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls") pod "image-registry-69fb5944f6-lff5w" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd") : secret "image-registry-tls" not found
Jul 01 19:21:58.919411 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.919376    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75cbd74cc9-tgr8g"]
Jul 01 19:21:58.952017 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.951993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:58.955170 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.955148    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 01 19:21:58.957108 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.957089    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 01 19:21:58.957229 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.957132    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zjpxm\""
Jul 01 19:21:58.960639 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.960621    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 01 19:21:58.960726 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.960702    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 01 19:21:58.962522 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.962495    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 01 19:21:58.962810 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:58.962797    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 01 19:21:59.002765 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.002746    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75cbd74cc9-tgr8g"]
Jul 01 19:21:59.016591 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.016571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.016682 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.016598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.016682 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.016614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x9jh\" (UniqueName: \"kubernetes.io/projected/cec1ec01-8cca-4ab5-a680-749611f0acb2-kube-api-access-4x9jh\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.016682 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.016631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-stats-auth\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.016786 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.016765    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-default-certificate\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.044149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.044121    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-47xvm"]
Jul 01 19:21:59.063745 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.063729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.068830 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.068804    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 01 19:21:59.069713 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.069687    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca-bundle\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap"
Jul 01 19:21:59.069798 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.069765    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-bundle\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap"
Jul 01 19:21:59.069840 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.069795    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"openshift-insights-serving-cert\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret"
Jul 01 19:21:59.075879 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.075861    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 01 19:21:59.078415 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.078397    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"]
Jul 01 19:21:59.089574 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.089554    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"operator-dockercfg-bfxbr\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bfxbr\"" type="*v1.Secret"
Jul 01 19:21:59.095788 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.095774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:21:59.100646 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.100621    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"telemetry-config\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap"
Jul 01 19:21:59.100792 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.100694    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-monitoring-operator-dockercfg-jrnx9\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jrnx9\"" type="*v1.Secret"
Jul 01 19:21:59.101946 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.101920    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-monitoring-operator-tls\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret"
Jul 01 19:21:59.102073 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.102057    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-47xvm"]
Jul 01 19:21:59.102640 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.102621    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 01 19:21:59.102742 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.102727    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-234.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-130-234.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 01 19:21:59.114438 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.114421    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"]
Jul 01 19:21:59.117814 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.117798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc491396-37b5-4232-ac52-26beccb0bd39-serving-cert\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.117905 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.117822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6p5\" (UniqueName: \"kubernetes.io/projected/4dbf254d-765f-48a8-941f-febf6e272995-kube-api-access-jl6p5\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:21:59.117905 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.117837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc491396-37b5-4232-ac52-26beccb0bd39-tmp\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.117905 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.117854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-stats-auth\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.117905 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.117871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.118066 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.117949    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:59.617937639 +0000 UTC m=+46.377464337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : configmap references non-existent config key: service-ca.crt
Jul 01 19:21:59.118066 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.117997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4dbf254d-765f-48a8-941f-febf6e272995-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:21:59.118066 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-service-ca-bundle\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.118066 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87f8m\" (UniqueName: \"kubernetes.io/projected/dc491396-37b5-4232-ac52-26beccb0bd39-kube-api-access-87f8m\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.118410 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-default-certificate\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.118539 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.121973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x9jh\" (UniqueName: \"kubernetes.io/projected/cec1ec01-8cca-4ab5-a680-749611f0acb2-kube-api-access-4x9jh\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.121973 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.118651    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 19:21:59.121973 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.118721    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:21:59.618704396 +0000 UTC m=+46.378231109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : secret "router-metrics-certs-default" not found
Jul 01 19:21:59.121973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dc491396-37b5-4232-ac52-26beccb0bd39-snapshots\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.121973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:21:59.121973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.118899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.123202 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.123183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-default-certificate\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.123793 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.123774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-stats-auth\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.150749 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.150724    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x9jh\" (UniqueName: \"kubernetes.io/projected/cec1ec01-8cca-4ab5-a680-749611f0acb2-kube-api-access-4x9jh\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.206768 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.206718    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"]
Jul 01 19:21:59.219933 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.219910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc491396-37b5-4232-ac52-26beccb0bd39-serving-cert\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.220022 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.219937    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6p5\" (UniqueName: \"kubernetes.io/projected/4dbf254d-765f-48a8-941f-febf6e272995-kube-api-access-jl6p5\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:21:59.220022 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.219953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc491396-37b5-4232-ac52-26beccb0bd39-tmp\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.220022 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.219988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4dbf254d-765f-48a8-941f-febf6e272995-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:21:59.220022 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.220004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-service-ca-bundle\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.220229 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.220025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87f8m\" (UniqueName: \"kubernetes.io/projected/dc491396-37b5-4232-ac52-26beccb0bd39-kube-api-access-87f8m\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.220229 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.220072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dc491396-37b5-4232-ac52-26beccb0bd39-snapshots\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.220229 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.220090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:21:59.220229 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.220106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.220429 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.220416    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc491396-37b5-4232-ac52-26beccb0bd39-tmp\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.220565 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.220548    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dc491396-37b5-4232-ac52-26beccb0bd39-snapshots\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.227850 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.227837    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"
Jul 01 19:21:59.234057 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.234039    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:59.234337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.234322    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7xhvr\""
Jul 01 19:21:59.236014 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.235905    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:59.236346 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.236331    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"]
Jul 01 19:21:59.249183 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.249161    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-pcww5"]
Jul 01 19:21:59.271950 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.271925    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-pcww5"]
Jul 01 19:21:59.272048 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.272000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5"
Jul 01 19:21:59.275010 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.274995    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7cr9d\""
Jul 01 19:21:59.321303 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.321275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhcb\" (UniqueName: \"kubernetes.io/projected/c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21-kube-api-access-bwhcb\") pod \"network-check-source-56499946bc-pcww5\" (UID: \"c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5"
Jul 01 19:21:59.321401 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.321381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfzz\" (UniqueName: \"kubernetes.io/projected/793deba2-438e-4029-a2bb-efdb8460962c-kube-api-access-4tfzz\") pod \"volume-data-source-validator-69d9f679db-q2bft\" (UID: \"793deba2-438e-4029-a2bb-efdb8460962c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"
Jul 01 19:21:59.330360 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.330337    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"]
Jul 01 19:21:59.347962 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.347887    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"]
Jul 01 19:21:59.348044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.347947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.356566 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.356551    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 01 19:21:59.356821 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.356809    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 01 19:21:59.356879 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.356839    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:59.357737 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.357719    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mgk55\""
Jul 01 19:21:59.358612 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.358595    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:59.373889 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.373870    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"]
Jul 01 19:21:59.374014 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.373996    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:21:59.385998 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.385975    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:59.386139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.385984    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:59.386139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.386006    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 01 19:21:59.386316 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.386301    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c4cj5\""
Jul 01 19:21:59.396086 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.396070    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"]
Jul 01 19:21:59.396206 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.396192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.399442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.399418    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 01 19:21:59.399542 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.399511    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fz96n\""
Jul 01 19:21:59.399739 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.399725    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:59.400013 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.399997    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:59.400086 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.400002    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 01 19:21:59.414149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.414133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"]
Jul 01 19:21:59.414249 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.414164    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"]
Jul 01 19:21:59.414249 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.414196    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"]
Jul 01 19:21:59.414249 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.414215    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"]
Jul 01 19:21:59.414402 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.414288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.419296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.418544    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 01 19:21:59.419902 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.419886    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 01 19:21:59.420164 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.420147    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ffh4b\""
Jul 01 19:21:59.420398 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.420384    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 01 19:21:59.421225 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 01 19:21:59.421658 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw88j\" (UniqueName: \"kubernetes.io/projected/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-kube-api-access-hw88j\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.421738 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.421738 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfzz\" (UniqueName: \"kubernetes.io/projected/793deba2-438e-4029-a2bb-efdb8460962c-kube-api-access-4tfzz\") pod \"volume-data-source-validator-69d9f679db-q2bft\" (UID: \"793deba2-438e-4029-a2bb-efdb8460962c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"
Jul 01 19:21:59.421738 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-config\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.421887 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:21:59.421887 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trv88\" (UniqueName: \"kubernetes.io/projected/f99ef790-7739-42c8-a8fe-5f9545943006-kube-api-access-trv88\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:21:59.421887 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/740904fe-42d6-4620-8421-048e8ffad910-config\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.421887 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnbn\" (UniqueName: \"kubernetes.io/projected/740904fe-42d6-4620-8421-048e8ffad910-kube-api-access-nlnbn\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.422082 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtgqn\" (UniqueName: \"kubernetes.io/projected/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-kube-api-access-vtgqn\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.422082 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.422082 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.421976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-serving-cert\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.422082 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.422070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhcb\" (UniqueName: \"kubernetes.io/projected/c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21-kube-api-access-bwhcb\") pod \"network-check-source-56499946bc-pcww5\" (UID: \"c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5"
Jul 01 19:21:59.422274 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.422145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/740904fe-42d6-4620-8421-048e8ffad910-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.422274 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.422172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-trusted-ca\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.426667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.426650    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 01 19:21:59.433512 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.433493    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhcb\" (UniqueName: \"kubernetes.io/projected/c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21-kube-api-access-bwhcb\") pod \"network-check-source-56499946bc-pcww5\" (UID: \"c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5"
Jul 01 19:21:59.433652 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.433637    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfzz\" (UniqueName: \"kubernetes.io/projected/793deba2-438e-4029-a2bb-efdb8460962c-kube-api-access-4tfzz\") pod \"volume-data-source-validator-69d9f679db-q2bft\" (UID: \"793deba2-438e-4029-a2bb-efdb8460962c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"
Jul 01 19:21:59.523005 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.522958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw88j\" (UniqueName: \"kubernetes.io/projected/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-kube-api-access-hw88j\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.523005 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.522987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.523142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-config\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.523142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:21:59.523142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trv88\" (UniqueName: \"kubernetes.io/projected/f99ef790-7739-42c8-a8fe-5f9545943006-kube-api-access-trv88\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:21:59.523142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/740904fe-42d6-4620-8421-048e8ffad910-config\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.523142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnbn\" (UniqueName: \"kubernetes.io/projected/740904fe-42d6-4620-8421-048e8ffad910-kube-api-access-nlnbn\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.523142 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.523109    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 19:21:59.523142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtgqn\" (UniqueName: \"kubernetes.io/projected/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-kube-api-access-vtgqn\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.523507 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.523172    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls podName:f99ef790-7739-42c8-a8fe-5f9545943006 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:00.023153328 +0000 UTC m=+46.782680043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-v6ns6" (UID: "f99ef790-7739-42c8-a8fe-5f9545943006") : secret "samples-operator-tls" not found
Jul 01 19:21:59.523507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.523507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-serving-cert\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.523507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/740904fe-42d6-4620-8421-048e8ffad910-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.523507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-trusted-ca\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.523798 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523773    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/740904fe-42d6-4620-8421-048e8ffad910-config\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.523857 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523793    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-config\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.523857 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.523842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-config\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.524345 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.524324    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-trusted-ca\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.525323 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.525297    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/740904fe-42d6-4620-8421-048e8ffad910-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.525463 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.525436    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-serving-cert\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.525599 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.525582    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-serving-cert\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.535980 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.535966    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"
Jul 01 19:21:59.538402 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.538376    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnbn\" (UniqueName: \"kubernetes.io/projected/740904fe-42d6-4620-8421-048e8ffad910-kube-api-access-nlnbn\") pod \"service-ca-operator-5d7b7dc95c-c74pn\" (UID: \"740904fe-42d6-4620-8421-048e8ffad910\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.538808 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.538792    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw88j\" (UniqueName: \"kubernetes.io/projected/d03eca50-fabd-4cd0-9a1c-465c4ee290c0-kube-api-access-hw88j\") pod \"kube-storage-version-migrator-operator-79964ff6f9-5fhht\" (UID: \"d03eca50-fabd-4cd0-9a1c-465c4ee290c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.538856 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.538822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trv88\" (UniqueName: \"kubernetes.io/projected/f99ef790-7739-42c8-a8fe-5f9545943006-kube-api-access-trv88\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:21:59.538856 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.538824    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtgqn\" (UniqueName: \"kubernetes.io/projected/a3b20d7f-4331-4dd8-88ed-bac6347f02e8-kube-api-access-vtgqn\") pod \"console-operator-7fdbdf9c99-pcrgm\" (UID: \"a3b20d7f-4331-4dd8-88ed-bac6347f02e8\") " pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.579505 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.579486    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5"
Jul 01 19:21:59.634799 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.634159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.634799 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.634248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:21:59.634799 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.634418    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 19:21:59.634799 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.634501    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:00.634481346 +0000 UTC m=+47.394008048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : secret "router-metrics-certs-default" not found
Jul 01 19:21:59.635088 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:21:59.634957    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:00.634938993 +0000 UTC m=+47.394465697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : configmap references non-existent config key: service-ca.crt
Jul 01 19:21:59.656479 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.656408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"
Jul 01 19:21:59.704397 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.704249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"
Jul 01 19:21:59.725492 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.725464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:21:59.740488 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.740442    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft"]
Jul 01 19:21:59.743701 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:59.743675    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793deba2_438e_4029_a2bb_efdb8460962c.slice/crio-6da27b225a2a9987bd038d9c9be7f9194e8497c5a16cb150005dcd06913d118b WatchSource:0}: Error finding container 6da27b225a2a9987bd038d9c9be7f9194e8497c5a16cb150005dcd06913d118b: Status 404 returned error can't find the container with id 6da27b225a2a9987bd038d9c9be7f9194e8497c5a16cb150005dcd06913d118b
Jul 01 19:21:59.751637 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:59.751438    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ea7a4d_5ae6_4cf4_9d1a_b4ff03c28b21.slice/crio-c1268d1c9c738989cbaea151440efc8a081dd7c53a153c75a27a1e4bab059600 WatchSource:0}: Error finding container c1268d1c9c738989cbaea151440efc8a081dd7c53a153c75a27a1e4bab059600: Status 404 returned error can't find the container with id c1268d1c9c738989cbaea151440efc8a081dd7c53a153c75a27a1e4bab059600
Jul 01 19:21:59.752328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.752252    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-pcww5"]
Jul 01 19:21:59.823477 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.823386    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn"]
Jul 01 19:21:59.827726 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:59.827687    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740904fe_42d6_4620_8421_048e8ffad910.slice/crio-2fd311bff7df19b3584c4a64534af5fffdf37cad4b95226224c4b76f31ae7d7a WatchSource:0}: Error finding container 2fd311bff7df19b3584c4a64534af5fffdf37cad4b95226224c4b76f31ae7d7a: Status 404 returned error can't find the container with id 2fd311bff7df19b3584c4a64534af5fffdf37cad4b95226224c4b76f31ae7d7a
Jul 01 19:21:59.883082 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.883061    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht"]
Jul 01 19:21:59.885210 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:59.885183    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03eca50_fabd_4cd0_9a1c_465c4ee290c0.slice/crio-b6a8d170894b5e6bd25b88ab1e2cb54699f0ed349ceaa58f278d9ebb3c57d12f WatchSource:0}: Error finding container b6a8d170894b5e6bd25b88ab1e2cb54699f0ed349ceaa58f278d9ebb3c57d12f: Status 404 returned error can't find the container with id b6a8d170894b5e6bd25b88ab1e2cb54699f0ed349ceaa58f278d9ebb3c57d12f
Jul 01 19:21:59.885311 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.885270    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 01 19:21:59.890613 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.890596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-service-ca-bundle\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:21:59.902851 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.902831    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"]
Jul 01 19:21:59.905222 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:21:59.905200    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b20d7f_4331_4dd8_88ed_bac6347f02e8.slice/crio-0a663c29e2930ee314f2dbbc7401940cda6be05ab8459543b1922f752b3371da WatchSource:0}: Error finding container 0a663c29e2930ee314f2dbbc7401940cda6be05ab8459543b1922f752b3371da: Status 404 returned error can't find the container with id 0a663c29e2930ee314f2dbbc7401940cda6be05ab8459543b1922f752b3371da
Jul 01 19:21:59.985670 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.985648    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 19:21:59.987822 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:21:59.987807    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jrnx9\""
Jul 01 19:22:00.038353 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.038330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:00.038491 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.038477    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 19:22:00.038545 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.038529    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls podName:f99ef790-7739-42c8-a8fe-5f9545943006 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:01.038515505 +0000 UTC m=+47.798042204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-v6ns6" (UID: "f99ef790-7739-42c8-a8fe-5f9545943006") : secret "samples-operator-tls" not found
Jul 01 19:22:00.059689 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.059665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn" event={"ID":"740904fe-42d6-4620-8421-048e8ffad910","Type":"ContainerStarted","Data":"2fd311bff7df19b3584c4a64534af5fffdf37cad4b95226224c4b76f31ae7d7a"}
Jul 01 19:22:00.060634 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.060615    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht" event={"ID":"d03eca50-fabd-4cd0-9a1c-465c4ee290c0","Type":"ContainerStarted","Data":"b6a8d170894b5e6bd25b88ab1e2cb54699f0ed349ceaa58f278d9ebb3c57d12f"}
Jul 01 19:22:00.061557 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.061535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft" event={"ID":"793deba2-438e-4029-a2bb-efdb8460962c","Type":"ContainerStarted","Data":"6da27b225a2a9987bd038d9c9be7f9194e8497c5a16cb150005dcd06913d118b"}
Jul 01 19:22:00.064414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.064380    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" event={"ID":"a3b20d7f-4331-4dd8-88ed-bac6347f02e8","Type":"ContainerStarted","Data":"0a663c29e2930ee314f2dbbc7401940cda6be05ab8459543b1922f752b3371da"}
Jul 01 19:22:00.065584 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.065566    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5" event={"ID":"c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21","Type":"ContainerStarted","Data":"c1268d1c9c738989cbaea151440efc8a081dd7c53a153c75a27a1e4bab059600"}
Jul 01 19:22:00.130686 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.130667    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 19:22:00.142808 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.142788    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6p5\" (UniqueName: \"kubernetes.io/projected/4dbf254d-765f-48a8-941f-febf6e272995-kube-api-access-jl6p5\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:00.189611 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.189585    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 01 19:22:00.191048 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.191023    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4dbf254d-765f-48a8-941f-febf6e272995-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:00.204030 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.204010    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 01 19:22:00.211051 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.211031    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:00.211152 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.211091    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls podName:4dbf254d-765f-48a8-941f-febf6e272995 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:00.711072486 +0000 UTC m=+47.470599185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-65jxg" (UID: "4dbf254d-765f-48a8-941f-febf6e272995") : secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:00.220370 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.220342    2576 configmap.go:193] Couldn't get configMap openshift-insights/trusted-ca-bundle: failed to sync configmap cache: timed out waiting for the condition
Jul 01 19:22:00.220486 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.220411    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-trusted-ca-bundle podName:dc491396-37b5-4232-ac52-26beccb0bd39 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:00.720394663 +0000 UTC m=+47.479921362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-trusted-ca-bundle") pod "insights-operator-5f67b55555-47xvm" (UID: "dc491396-37b5-4232-ac52-26beccb0bd39") : failed to sync configmap cache: timed out waiting for the condition
Jul 01 19:22:00.220486 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.220348    2576 secret.go:189] Couldn't get secret openshift-insights/openshift-insights-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 01 19:22:00.220486 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.220476    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc491396-37b5-4232-ac52-26beccb0bd39-serving-cert podName:dc491396-37b5-4232-ac52-26beccb0bd39 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:00.720465696 +0000 UTC m=+47.479992397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/dc491396-37b5-4232-ac52-26beccb0bd39-serving-cert") pod "insights-operator-5f67b55555-47xvm" (UID: "dc491396-37b5-4232-ac52-26beccb0bd39") : failed to sync secret cache: timed out waiting for the condition
Jul 01 19:22:00.227682 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.227659    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 01 19:22:00.233603 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.233579    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 19:22:00.296927 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.296870    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 19:22:00.304961 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.304939    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87f8m\" (UniqueName: \"kubernetes.io/projected/dc491396-37b5-4232-ac52-26beccb0bd39-kube-api-access-87f8m\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:22:00.410231 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.410203    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"]
Jul 01 19:22:00.423022 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.423001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:00.426383 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.426354    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"]
Jul 01 19:22:00.426489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.426356    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q2wc6\""
Jul 01 19:22:00.428257 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.428229    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 01 19:22:00.428594 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.428572    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 01 19:22:00.442536 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.442511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:00.442710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.442690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a117e56-ec3c-4923-97b3-a0425bcdae19-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:00.500990 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.500967    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bfxbr\""
Jul 01 19:22:00.523537 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.523500    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 01 19:22:00.544139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.543534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a117e56-ec3c-4923-97b3-a0425bcdae19-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:00.544139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.543684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:00.544139 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.543890    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:22:00.544139 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.543948    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert podName:1a117e56-ec3c-4923-97b3-a0425bcdae19 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:01.043929996 +0000 UTC m=+47.803456701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-xcw24" (UID: "1a117e56-ec3c-4923-97b3-a0425bcdae19") : secret "networking-console-plugin-cert" not found
Jul 01 19:22:00.544444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.544339    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a117e56-ec3c-4923-97b3-a0425bcdae19-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:00.644947 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.644189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:00.644947 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.644281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:00.644947 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.644470    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 19:22:00.644947 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.644530    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:02.64451184 +0000 UTC m=+49.404038547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : secret "router-metrics-certs-default" not found
Jul 01 19:22:00.644947 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.644600    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:02.644589671 +0000 UTC m=+49.404116374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : configmap references non-existent config key: service-ca.crt
Jul 01 19:22:00.746006 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.745420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:00.746006 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.745483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:22:00.746006 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.745565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc491396-37b5-4232-ac52-26beccb0bd39-serving-cert\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:22:00.746006 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.745872    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:00.746006 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:00.745944    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls podName:4dbf254d-765f-48a8-941f-febf6e272995 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:01.745924069 +0000 UTC m=+48.505450771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-65jxg" (UID: "4dbf254d-765f-48a8-941f-febf6e272995") : secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:00.746736 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.746686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc491396-37b5-4232-ac52-26beccb0bd39-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:22:00.749582 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.749558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc491396-37b5-4232-ac52-26beccb0bd39-serving-cert\") pod \"insights-operator-5f67b55555-47xvm\" (UID: \"dc491396-37b5-4232-ac52-26beccb0bd39\") " pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:22:00.872125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:00.871858    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-47xvm"
Jul 01 19:22:01.048785 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:01.048753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:01.049181 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:01.048877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:01.049181 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:01.049089    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:22:01.049181 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:01.049149    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert podName:1a117e56-ec3c-4923-97b3-a0425bcdae19 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:02.049130094 +0000 UTC m=+48.808656795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-xcw24" (UID: "1a117e56-ec3c-4923-97b3-a0425bcdae19") : secret "networking-console-plugin-cert" not found
Jul 01 19:22:01.049481 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:01.049439    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 19:22:01.049558 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:01.049524    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls podName:f99ef790-7739-42c8-a8fe-5f9545943006 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:03.049506562 +0000 UTC m=+49.809033276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-v6ns6" (UID: "f99ef790-7739-42c8-a8fe-5f9545943006") : secret "samples-operator-tls" not found
Jul 01 19:22:01.059791 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:01.059648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-47xvm"]
Jul 01 19:22:01.582654 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:01.582618    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc491396_37b5_4232_ac52_26beccb0bd39.slice/crio-4ea405f15f3883449a92c370a71f9c73cfc8e2860c34f7f711ced094d1d71da1 WatchSource:0}: Error finding container 4ea405f15f3883449a92c370a71f9c73cfc8e2860c34f7f711ced094d1d71da1: Status 404 returned error can't find the container with id 4ea405f15f3883449a92c370a71f9c73cfc8e2860c34f7f711ced094d1d71da1
Jul 01 19:22:01.758116 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:01.758025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:01.758410 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:01.758378    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:01.758552 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:01.758466    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls podName:4dbf254d-765f-48a8-941f-febf6e272995 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:03.758432935 +0000 UTC m=+50.517959634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-65jxg" (UID: "4dbf254d-765f-48a8-941f-febf6e272995") : secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:02.061244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:02.061211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:02.061722 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:02.061394    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:22:02.061722 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:02.061489    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert podName:1a117e56-ec3c-4923-97b3-a0425bcdae19 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:04.06146767 +0000 UTC m=+50.820994371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-xcw24" (UID: "1a117e56-ec3c-4923-97b3-a0425bcdae19") : secret "networking-console-plugin-cert" not found
Jul 01 19:22:02.074870 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:02.074835    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-47xvm" event={"ID":"dc491396-37b5-4232-ac52-26beccb0bd39","Type":"ContainerStarted","Data":"4ea405f15f3883449a92c370a71f9c73cfc8e2860c34f7f711ced094d1d71da1"}
Jul 01 19:22:02.667257 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:02.667224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:02.667387 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:02.667357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:02.667468 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:02.667381    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 19:22:02.667468 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:02.667445    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:06.667427589 +0000 UTC m=+53.426954299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : secret "router-metrics-certs-default" not found
Jul 01 19:22:02.667577 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:02.667554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:06.66753286 +0000 UTC m=+53.427059570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : configmap references non-existent config key: service-ca.crt
Jul 01 19:22:03.070719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:03.070686    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:03.071128 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.070812    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 19:22:03.071128 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.070877    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls podName:f99ef790-7739-42c8-a8fe-5f9545943006 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:07.07086228 +0000 UTC m=+53.830388979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-v6ns6" (UID: "f99ef790-7739-42c8-a8fe-5f9545943006") : secret "samples-operator-tls" not found
Jul 01 19:22:03.777543 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:03.777504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:03.777686 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.777633    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:03.777743 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.777702    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls podName:4dbf254d-765f-48a8-941f-febf6e272995 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:07.777681063 +0000 UTC m=+54.537207762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-65jxg" (UID: "4dbf254d-765f-48a8-941f-febf6e272995") : secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:03.878333 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:03.878296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:22:03.878510 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:03.878364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:22:03.878510 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.878471    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 19:22:03.878510 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.878489    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69fb5944f6-lff5w: secret "image-registry-tls" not found
Jul 01 19:22:03.878662 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.878538    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 19:22:03.878662 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.878553    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls podName:62e6f8b2-4af9-4576-b751-aa4176320cbd nodeName:}" failed. No retries permitted until 2026-07-01 19:22:19.878533685 +0000 UTC m=+66.638060383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls") pod "image-registry-69fb5944f6-lff5w" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd") : secret "image-registry-tls" not found
Jul 01 19:22:03.878662 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:03.878596    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls podName:0d95f075-ee0e-498d-8cb0-52a0d3080e5c nodeName:}" failed. No retries permitted until 2026-07-01 19:22:19.878578992 +0000 UTC m=+66.638105709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls") pod "dns-default-ch6qk" (UID: "0d95f075-ee0e-498d-8cb0-52a0d3080e5c") : secret "dns-default-metrics-tls" not found
Jul 01 19:22:04.080561 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:04.080521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:04.081017 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:04.080680    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:22:04.081017 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:04.080771    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert podName:1a117e56-ec3c-4923-97b3-a0425bcdae19 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:08.080749304 +0000 UTC m=+54.840276018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-xcw24" (UID: "1a117e56-ec3c-4923-97b3-a0425bcdae19") : secret "networking-console-plugin-cert" not found
Jul 01 19:22:06.086904 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:06.086829    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5" event={"ID":"c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21","Type":"ContainerStarted","Data":"e9206ac73c751fd84003eb8292d5ddb3e0a9db356bb72d26454951493a801142"}
Jul 01 19:22:06.088789 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:06.088766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft" event={"ID":"793deba2-438e-4029-a2bb-efdb8460962c","Type":"ContainerStarted","Data":"6d1418b25d6ecfca87f645762d1c2f96aac6deece9ac21c59648c02a82189591"}
Jul 01 19:22:06.702399 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:06.702364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:06.702583 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:06.702529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:06.702583 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:06.702553    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 19:22:06.702685 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:06.702622    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:14.702602912 +0000 UTC m=+61.462129624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : secret "router-metrics-certs-default" not found
Jul 01 19:22:06.702743 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:06.702703    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:14.702685472 +0000 UTC m=+61.462212177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : configmap references non-existent config key: service-ca.crt
Jul 01 19:22:07.094665 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.094630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht" event={"ID":"d03eca50-fabd-4cd0-9a1c-465c4ee290c0","Type":"ContainerStarted","Data":"577cc16065809468e7c6e46d0c9a14c4c22dc33c08cf04a9d68ab3cf9ab59acd"}
Jul 01 19:22:07.096433 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.096412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/0.log"
Jul 01 19:22:07.096571 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.096471    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8" containerID="9c4ef90a42297c3bed08c3366241e7b1814682dbd4a9e3502a2c954747f85d7f" exitCode=255
Jul 01 19:22:07.096571 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.096559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" event={"ID":"a3b20d7f-4331-4dd8-88ed-bac6347f02e8","Type":"ContainerDied","Data":"9c4ef90a42297c3bed08c3366241e7b1814682dbd4a9e3502a2c954747f85d7f"}
Jul 01 19:22:07.096732 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.096713    2576 scope.go:117] "RemoveContainer" containerID="9c4ef90a42297c3bed08c3366241e7b1814682dbd4a9e3502a2c954747f85d7f"
Jul 01 19:22:07.098393 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.098195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-47xvm" event={"ID":"dc491396-37b5-4232-ac52-26beccb0bd39","Type":"ContainerStarted","Data":"0f0116a87ea147b82b1018d32232d6452d694ccd0d609dbf51fe2e9b5b5e54f4"}
Jul 01 19:22:07.100487 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.100462    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn" event={"ID":"740904fe-42d6-4620-8421-048e8ffad910","Type":"ContainerStarted","Data":"958112ce1313c919ff7127e883b76e24cf45e77b5eb7aa0dedf42ff283728464"}
Jul 01 19:22:07.105942 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.105915    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:07.106079 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:07.106062    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 19:22:07.106140 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:07.106132    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls podName:f99ef790-7739-42c8-a8fe-5f9545943006 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:15.106113857 +0000 UTC m=+61.865640562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-v6ns6" (UID: "f99ef790-7739-42c8-a8fe-5f9545943006") : secret "samples-operator-tls" not found
Jul 01 19:22:07.117711 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.117665    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht" podStartSLOduration=2.05144753 podStartE2EDuration="8.117648565s" totalImagesPullingTime="6.066201035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:07.116472681 +0000 UTC m=+53.875999394" watchObservedRunningTime="2026-07-01 19:22:07.117648565 +0000 UTC m=+53.877175287"
Jul 01 19:22:07.142858 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.142808    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-q2bft" podStartSLOduration=2.515284189 podStartE2EDuration="8.14279123s" totalImagesPullingTime="5.627507041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:07.141265722 +0000 UTC m=+53.900792444" watchObservedRunningTime="2026-07-01 19:22:07.14279123 +0000 UTC m=+53.902317953"
Jul 01 19:22:07.189742 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.189688    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-pcww5" podStartSLOduration=1.989750237 podStartE2EDuration="8.189669602s" totalImagesPullingTime="6.199919365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:07.189081637 +0000 UTC m=+53.948608354" watchObservedRunningTime="2026-07-01 19:22:07.189669602 +0000 UTC m=+53.949196318"
Jul 01 19:22:07.221018 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.220964    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn" podStartSLOduration=2.097842523 podStartE2EDuration="8.220947056s" totalImagesPullingTime="6.123104533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:07.219896396 +0000 UTC m=+53.979423115" watchObservedRunningTime="2026-07-01 19:22:07.220947056 +0000 UTC m=+53.980473776"
Jul 01 19:22:07.813042 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:07.813009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:07.813267 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:07.813118    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:07.813267 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:07.813168    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls podName:4dbf254d-765f-48a8-941f-febf6e272995 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:15.813155122 +0000 UTC m=+62.572681824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-65jxg" (UID: "4dbf254d-765f-48a8-941f-febf6e272995") : secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:08.105091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.105024    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/1.log"
Jul 01 19:22:08.105495 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.105432    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/0.log"
Jul 01 19:22:08.105558 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.105495    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8" containerID="b4c8c291d76e1a1b0d8ea05991af045849e8adcdc7723444cb38a4f8a4a2da3b" exitCode=255
Jul 01 19:22:08.105616 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.105581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" event={"ID":"a3b20d7f-4331-4dd8-88ed-bac6347f02e8","Type":"ContainerDied","Data":"b4c8c291d76e1a1b0d8ea05991af045849e8adcdc7723444cb38a4f8a4a2da3b"}
Jul 01 19:22:08.105664 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.105654    2576 scope.go:117] "RemoveContainer" containerID="9c4ef90a42297c3bed08c3366241e7b1814682dbd4a9e3502a2c954747f85d7f"
Jul 01 19:22:08.105796 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.105775    2576 scope.go:117] "RemoveContainer" containerID="b4c8c291d76e1a1b0d8ea05991af045849e8adcdc7723444cb38a4f8a4a2da3b"
Jul 01 19:22:08.106073 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:08.106032    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-pcrgm_openshift-console-operator(a3b20d7f-4331-4dd8-88ed-bac6347f02e8)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" podUID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8"
Jul 01 19:22:08.114993 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.114970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:08.115144 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:08.115126    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:22:08.115195 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:08.115190    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert podName:1a117e56-ec3c-4923-97b3-a0425bcdae19 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:16.115175586 +0000 UTC m=+62.874702285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-xcw24" (UID: "1a117e56-ec3c-4923-97b3-a0425bcdae19") : secret "networking-console-plugin-cert" not found
Jul 01 19:22:08.127257 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.127221    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-47xvm" podStartSLOduration=4.323795756 podStartE2EDuration="9.127209501s" totalImagesPullingTime="4.803413745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:07.253578256 +0000 UTC m=+54.013104979" watchObservedRunningTime="2026-07-01 19:22:08.127209501 +0000 UTC m=+54.886736222"
Jul 01 19:22:08.447784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.447706    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m5nxr"]
Jul 01 19:22:08.450769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.450746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.456558 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.456543    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 19:22:08.456798 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.456778    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tz26f\""
Jul 01 19:22:08.456798 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.456790    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 19:22:08.469485 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.469423    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m5nxr"]
Jul 01 19:22:08.518768 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.518748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.518872 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.518775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqtv7\" (UniqueName: \"kubernetes.io/projected/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-api-access-wqtv7\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.518872 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.518814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-crio-socket\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.518872 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.518846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.518967 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.518884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-data-volume\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.619749 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.619724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.619884 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.619755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtv7\" (UniqueName: \"kubernetes.io/projected/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-api-access-wqtv7\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.619884 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.619789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-crio-socket\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.619884 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.619814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.619884 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:08.619846    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:08.619884 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.619866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-crio-socket\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.620042 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:08.619892    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:08.620042 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:08.619913    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:09.119897245 +0000 UTC m=+55.879423958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:08.620042 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:08.619939    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:09.119922852 +0000 UTC m=+55.879449567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:08.620042 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.620006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-data-volume\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.620308 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.620292    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-data-volume\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:08.631511 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:08.631492    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtv7\" (UniqueName: \"kubernetes.io/projected/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-api-access-wqtv7\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:09.109970 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:09.109946    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/1.log"
Jul 01 19:22:09.110348 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:09.110302    2576 scope.go:117] "RemoveContainer" containerID="b4c8c291d76e1a1b0d8ea05991af045849e8adcdc7723444cb38a4f8a4a2da3b"
Jul 01 19:22:09.110516 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:09.110494    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-pcrgm_openshift-console-operator(a3b20d7f-4331-4dd8-88ed-bac6347f02e8)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" podUID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8"
Jul 01 19:22:09.124219 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:09.124193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:09.124317 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:09.124265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:09.124317 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:09.124300    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:09.124417 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:09.124401    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:09.124480 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:09.124439    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:10.124421792 +0000 UTC m=+56.883948497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:09.124480 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:09.124471    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:10.12446161 +0000 UTC m=+56.883988316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:09.726060 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:09.726022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:22:09.726289 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:09.726082    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:22:10.112873 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:10.112847    2576 scope.go:117] "RemoveContainer" containerID="b4c8c291d76e1a1b0d8ea05991af045849e8adcdc7723444cb38a4f8a4a2da3b"
Jul 01 19:22:10.113217 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:10.113007    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-pcrgm_openshift-console-operator(a3b20d7f-4331-4dd8-88ed-bac6347f02e8)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" podUID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8"
Jul 01 19:22:10.133116 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:10.133086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:10.133261 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:10.133137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:10.133261 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:10.133200    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:10.133261 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:10.133219    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:10.133361 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:10.133279    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:12.133264863 +0000 UTC m=+58.892791563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:10.133361 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:10.133294    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:12.133288645 +0000 UTC m=+58.892815343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:11.214636 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:11.214607    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mg6xr_cb7ee79e-263e-4f31-a115-9374f163a704/dns-node-resolver/0.log"
Jul 01 19:22:12.149194 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:12.149161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:12.149383 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:12.149213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:12.149383 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:12.149291    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:12.149383 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:12.149322    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:12.149383 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:12.149363    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:16.149344959 +0000 UTC m=+62.908871662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:12.149383 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:12.149381    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:16.149372841 +0000 UTC m=+62.908899554 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:12.214271 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:12.214250    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gc59t_6d88f24b-68e6-4d58-ac3e-98f1c68a55be/node-ca/0.log"
Jul 01 19:22:12.816809 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:12.816781    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2mnld_b55a5e16-7871-451f-a5c6-be4314ca7fb5/serve-healthcheck-canary/0.log"
Jul 01 19:22:13.018832 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:13.018806    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sxlfd"
Jul 01 19:22:13.815207 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:13.815183    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-5fhht_d03eca50-fabd-4cd0-9a1c-465c4ee290c0/kube-storage-version-migrator-operator/0.log"
Jul 01 19:22:14.772323 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:14.772288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:14.772784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:14.772354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:14.772784 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:14.772484    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 19:22:14.772784 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:14.772508    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:30.772480934 +0000 UTC m=+77.532007635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : configmap references non-existent config key: service-ca.crt
Jul 01 19:22:14.772784 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:14.772627    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs podName:cec1ec01-8cca-4ab5-a680-749611f0acb2 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:30.772611728 +0000 UTC m=+77.532138428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs") pod "router-default-75cbd74cc9-tgr8g" (UID: "cec1ec01-8cca-4ab5-a680-749611f0acb2") : secret "router-metrics-certs-default" not found
Jul 01 19:22:15.176173 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:15.176142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:15.176362 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:15.176320    2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 01 19:22:15.176432 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:15.176397    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls podName:f99ef790-7739-42c8-a8fe-5f9545943006 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:31.176375191 +0000 UTC m=+77.935901905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-v6ns6" (UID: "f99ef790-7739-42c8-a8fe-5f9545943006") : secret "samples-operator-tls" not found
Jul 01 19:22:15.882170 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:15.882143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:15.882555 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:15.882282    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:15.882555 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:15.882339    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls podName:4dbf254d-765f-48a8-941f-febf6e272995 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:31.882324922 +0000 UTC m=+78.641851621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-65jxg" (UID: "4dbf254d-765f-48a8-941f-febf6e272995") : secret "cluster-monitoring-operator-tls" not found
Jul 01 19:22:16.184592 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:16.184499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:16.184757 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:16.184602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:16.184757 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:16.184642    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 01 19:22:16.184757 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:16.184654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:16.184757 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:16.184701    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert podName:1a117e56-ec3c-4923-97b3-a0425bcdae19 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:32.184684709 +0000 UTC m=+78.944211410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert") pod "networking-console-plugin-6cdb5968b9-xcw24" (UID: "1a117e56-ec3c-4923-97b3-a0425bcdae19") : secret "networking-console-plugin-cert" not found
Jul 01 19:22:16.184757 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:16.184716    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 19:22:16.184757 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:16.184742    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:16.185011 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:16.184786    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:24.184772799 +0000 UTC m=+70.944299498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:16.185011 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:16.184836    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:24.18482544 +0000 UTC m=+70.944352143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : configmap "kube-rbac-proxy" not found
Jul 01 19:22:18.504741 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.504707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:22:18.507344 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.507327    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 19:22:18.515572 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:18.515546    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 19:22:18.515679 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:18.515606    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs podName:a22a2075-444a-40ee-8406-7cd45cd55403 nodeName:}" failed. No retries permitted until 2026-07-01 19:23:22.515592082 +0000 UTC m=+129.275118784 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs") pod "network-metrics-daemon-r2hbh" (UID: "a22a2075-444a-40ee-8406-7cd45cd55403") : secret "metrics-daemon-secret" not found
Jul 01 19:22:18.605065 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.605041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:22:18.605161 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.605085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:22:18.607553 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.607528    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96vl\" (UniqueName: \"kubernetes.io/projected/1e5dcab4-31fb-4c7f-a51e-5dc3a0792059-kube-api-access-b96vl\") pod \"network-check-target-q4qsh\" (UID: \"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059\") " pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:22:18.607855 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.607841    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 19:22:18.618033 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.618014    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c8d1dd1d-8578-45a0-8f2a-cecb9128bb34-original-pull-secret\") pod \"global-pull-secret-syncer-nfh7p\" (UID: \"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34\") " pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:22:18.757161 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.757101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nfh7p"
Jul 01 19:22:18.861335 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.858644    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zbvbl\""
Jul 01 19:22:18.866707 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.866684    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:22:18.874229 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.874207    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nfh7p"]
Jul 01 19:22:18.877045 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:18.877021    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d1dd1d_8578_45a0_8f2a_cecb9128bb34.slice/crio-fe98f7311ac5e5e228fdb30d1c7307b1ce0b4b11a624abe01b78ebcd4375fc1d WatchSource:0}: Error finding container fe98f7311ac5e5e228fdb30d1c7307b1ce0b4b11a624abe01b78ebcd4375fc1d: Status 404 returned error can't find the container with id fe98f7311ac5e5e228fdb30d1c7307b1ce0b4b11a624abe01b78ebcd4375fc1d
Jul 01 19:22:18.985784 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:18.985702    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q4qsh"]
Jul 01 19:22:18.987769 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:18.987740    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5dcab4_31fb_4c7f_a51e_5dc3a0792059.slice/crio-92b7cb7f3561524f8712366546413cce3aeee86a0dbdf39fb3c30d242e85b0f5 WatchSource:0}: Error finding container 92b7cb7f3561524f8712366546413cce3aeee86a0dbdf39fb3c30d242e85b0f5: Status 404 returned error can't find the container with id 92b7cb7f3561524f8712366546413cce3aeee86a0dbdf39fb3c30d242e85b0f5
Jul 01 19:22:19.140966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.140932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4qsh" event={"ID":"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059","Type":"ContainerStarted","Data":"2c723f72023363b4f19f6ee546ae7afdeea027fe9470c3ae8eb5677da844a441"}
Jul 01 19:22:19.141128 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.140979    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q4qsh" event={"ID":"1e5dcab4-31fb-4c7f-a51e-5dc3a0792059","Type":"ContainerStarted","Data":"92b7cb7f3561524f8712366546413cce3aeee86a0dbdf39fb3c30d242e85b0f5"}
Jul 01 19:22:19.141128 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.141059    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:22:19.142073 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.142053    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nfh7p" event={"ID":"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34","Type":"ContainerStarted","Data":"fe98f7311ac5e5e228fdb30d1c7307b1ce0b4b11a624abe01b78ebcd4375fc1d"}
Jul 01 19:22:19.165179 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.165134    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q4qsh" podStartSLOduration=66.16512151 podStartE2EDuration="1m6.16512151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:19.164203695 +0000 UTC m=+65.923730407" watchObservedRunningTime="2026-07-01 19:22:19.16512151 +0000 UTC m=+65.924648230"
Jul 01 19:22:19.917355 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.917324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:22:19.917761 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.917375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:22:19.920173 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.920147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d95f075-ee0e-498d-8cb0-52a0d3080e5c-metrics-tls\") pod \"dns-default-ch6qk\" (UID: \"0d95f075-ee0e-498d-8cb0-52a0d3080e5c\") " pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:22:19.920380 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:19.920358    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"image-registry-69fb5944f6-lff5w\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") " pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:22:20.145571 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:20.145539    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wklt5\""
Jul 01 19:22:20.153090 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:20.153065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:22:20.166837 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:20.166785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f8prn\""
Jul 01 19:22:20.174389 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:20.174328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:22:20.316597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:20.316516    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69fb5944f6-lff5w"]
Jul 01 19:22:20.319361 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:20.319335    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e6f8b2_4af9_4576_b751_aa4176320cbd.slice/crio-565f11038174323847b6516223998219be9931a6237490a5db719ca4463e935a WatchSource:0}: Error finding container 565f11038174323847b6516223998219be9931a6237490a5db719ca4463e935a: Status 404 returned error can't find the container with id 565f11038174323847b6516223998219be9931a6237490a5db719ca4463e935a
Jul 01 19:22:20.331812 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:20.331742    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ch6qk"]
Jul 01 19:22:20.337013 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:20.336992    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d95f075_ee0e_498d_8cb0_52a0d3080e5c.slice/crio-43b53fcbddd7019b687eca6a6ac2fda5a4fafb1f94cdf5890fee25d4ef0ec6bb WatchSource:0}: Error finding container 43b53fcbddd7019b687eca6a6ac2fda5a4fafb1f94cdf5890fee25d4ef0ec6bb: Status 404 returned error can't find the container with id 43b53fcbddd7019b687eca6a6ac2fda5a4fafb1f94cdf5890fee25d4ef0ec6bb
Jul 01 19:22:21.151957 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:21.151834    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w" event={"ID":"62e6f8b2-4af9-4576-b751-aa4176320cbd","Type":"ContainerStarted","Data":"a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b"}
Jul 01 19:22:21.151957 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:21.151915    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:22:21.151957 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:21.151937    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w" event={"ID":"62e6f8b2-4af9-4576-b751-aa4176320cbd","Type":"ContainerStarted","Data":"565f11038174323847b6516223998219be9931a6237490a5db719ca4463e935a"}
Jul 01 19:22:21.153410 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:21.153374    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ch6qk" event={"ID":"0d95f075-ee0e-498d-8cb0-52a0d3080e5c","Type":"ContainerStarted","Data":"43b53fcbddd7019b687eca6a6ac2fda5a4fafb1f94cdf5890fee25d4ef0ec6bb"}
Jul 01 19:22:21.174961 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:21.174843    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w" podStartSLOduration=67.174821875 podStartE2EDuration="1m7.174821875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:21.173815418 +0000 UTC m=+67.933342144" watchObservedRunningTime="2026-07-01 19:22:21.174821875 +0000 UTC m=+67.934348594"
Jul 01 19:22:24.164483 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.164428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nfh7p" event={"ID":"c8d1dd1d-8578-45a0-8f2a-cecb9128bb34","Type":"ContainerStarted","Data":"93c274141579516489a67615a269d4a637b73ba2a42ea66756ce8bc778d7ef6d"}
Jul 01 19:22:24.166151 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.166127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ch6qk" event={"ID":"0d95f075-ee0e-498d-8cb0-52a0d3080e5c","Type":"ContainerStarted","Data":"44afa18b9fc58b87a4985b18ea702387049d333f0d2090f3be0097ec652084f9"}
Jul 01 19:22:24.166297 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.166166    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ch6qk" event={"ID":"0d95f075-ee0e-498d-8cb0-52a0d3080e5c","Type":"ContainerStarted","Data":"13a757f88a6c4f841392235ef43d0b45153c3328fcb6cc978a8d0d5932c9238c"}
Jul 01 19:22:24.166297 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.166267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:22:24.181022 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.180981    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nfh7p" podStartSLOduration=65.369512695 podStartE2EDuration="1m10.18096793s" totalImagesPullingTime="4.811455235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:24.180363936 +0000 UTC m=+70.939890657" watchObservedRunningTime="2026-07-01 19:22:24.18096793 +0000 UTC m=+70.940494650"
Jul 01 19:22:24.200179 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.200143    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ch6qk" podStartSLOduration=32.860061497 podStartE2EDuration="36.200130712s" totalImagesPullingTime="3.340069215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:24.199518222 +0000 UTC m=+70.959044944" watchObservedRunningTime="2026-07-01 19:22:24.200130712 +0000 UTC m=+70.959657432"
Jul 01 19:22:24.260336 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.260313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:24.260652 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.260629    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:24.260786 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:24.260768    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:24.260866 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:24.260841    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls podName:cf0ccf0b-f984-4bcd-8046-8e11b82d19e0 nodeName:}" failed. No retries permitted until 2026-07-01 19:22:40.260825264 +0000 UTC m=+87.020351967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m5nxr" (UID: "cf0ccf0b-f984-4bcd-8046-8e11b82d19e0") : secret "insights-runtime-extractor-tls" not found
Jul 01 19:22:24.260925 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.260865    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:24.744743 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:24.744712    2576 scope.go:117] "RemoveContainer" containerID="b4c8c291d76e1a1b0d8ea05991af045849e8adcdc7723444cb38a4f8a4a2da3b"
Jul 01 19:22:25.170815 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:25.170791    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:22:25.171200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:25.171184    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/1.log"
Jul 01 19:22:25.171247 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:25.171215    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8" containerID="1b3d0fae7adcda87d8b608d7903618dc20aa3ad2765a7c0dc64f44da70876005" exitCode=255
Jul 01 19:22:25.171321 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:25.171301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" event={"ID":"a3b20d7f-4331-4dd8-88ed-bac6347f02e8","Type":"ContainerDied","Data":"1b3d0fae7adcda87d8b608d7903618dc20aa3ad2765a7c0dc64f44da70876005"}
Jul 01 19:22:25.171374 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:25.171361    2576 scope.go:117] "RemoveContainer" containerID="b4c8c291d76e1a1b0d8ea05991af045849e8adcdc7723444cb38a4f8a4a2da3b"
Jul 01 19:22:25.171857 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:25.171841    2576 scope.go:117] "RemoveContainer" containerID="1b3d0fae7adcda87d8b608d7903618dc20aa3ad2765a7c0dc64f44da70876005"
Jul 01 19:22:25.172016 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:25.171999    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-pcrgm_openshift-console-operator(a3b20d7f-4331-4dd8-88ed-bac6347f02e8)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" podUID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8"
Jul 01 19:22:26.175742 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:26.175712    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:22:29.726277 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:29.726248    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:22:29.726664 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:29.726292    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:22:29.726664 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:29.726564    2576 scope.go:117] "RemoveContainer" containerID="1b3d0fae7adcda87d8b608d7903618dc20aa3ad2765a7c0dc64f44da70876005"
Jul 01 19:22:29.726737 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:29.726713    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-pcrgm_openshift-console-operator(a3b20d7f-4331-4dd8-88ed-bac6347f02e8)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" podUID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8"
Jul 01 19:22:30.810391 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:30.810357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:30.810752 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:30.810413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:30.810976 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:30.810955    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cec1ec01-8cca-4ab5-a680-749611f0acb2-service-ca-bundle\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:30.812581 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:30.812563    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cec1ec01-8cca-4ab5-a680-749611f0acb2-metrics-certs\") pod \"router-default-75cbd74cc9-tgr8g\" (UID: \"cec1ec01-8cca-4ab5-a680-749611f0acb2\") " pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:31.063346 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.063267    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zjpxm\""
Jul 01 19:22:31.070827 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.070805    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:31.213017 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.212987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:31.215694 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.215660    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f99ef790-7739-42c8-a8fe-5f9545943006-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-v6ns6\" (UID: \"f99ef790-7739-42c8-a8fe-5f9545943006\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:31.216441 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.216420    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75cbd74cc9-tgr8g"]
Jul 01 19:22:31.218691 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:31.218666    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec1ec01_8cca_4ab5_a680_749611f0acb2.slice/crio-c104a6a5b18353d053651bb6de317526e24f6b0e02d8f84f204cc7c9d8b5172f WatchSource:0}: Error finding container c104a6a5b18353d053651bb6de317526e24f6b0e02d8f84f204cc7c9d8b5172f: Status 404 returned error can't find the container with id c104a6a5b18353d053651bb6de317526e24f6b0e02d8f84f204cc7c9d8b5172f
Jul 01 19:22:31.485542 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.485470    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c4cj5\""
Jul 01 19:22:31.492719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.492701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"
Jul 01 19:22:31.644750 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.644716    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6"]
Jul 01 19:22:31.920708 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.920679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:31.923069 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:31.923039    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dbf254d-765f-48a8-941f-febf6e272995-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-65jxg\" (UID: \"4dbf254d-765f-48a8-941f-febf6e272995\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:32.106813 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.106784    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jrnx9\""
Jul 01 19:22:32.114921 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.114905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"
Jul 01 19:22:32.198272 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.198241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6" event={"ID":"f99ef790-7739-42c8-a8fe-5f9545943006","Type":"ContainerStarted","Data":"84416da93fbbf9e6c99ab3c75a146360a73fbd147e2d41126a2e93e31e2f2c13"}
Jul 01 19:22:32.199870 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.199842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g" event={"ID":"cec1ec01-8cca-4ab5-a680-749611f0acb2","Type":"ContainerStarted","Data":"49d09c399e43a444b92c6189f18fc10eb85f25c9dac5ec6de7e14fcf8fa20b73"}
Jul 01 19:22:32.200003 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.199888    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g" event={"ID":"cec1ec01-8cca-4ab5-a680-749611f0acb2","Type":"ContainerStarted","Data":"c104a6a5b18353d053651bb6de317526e24f6b0e02d8f84f204cc7c9d8b5172f"}
Jul 01 19:22:32.222520 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.222495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:32.224686 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.224665    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a117e56-ec3c-4923-97b3-a0425bcdae19-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-xcw24\" (UID: \"1a117e56-ec3c-4923-97b3-a0425bcdae19\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:32.240841 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.240819    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-q2wc6\""
Jul 01 19:22:32.242582 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.242544    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g" podStartSLOduration=34.242531519 podStartE2EDuration="34.242531519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:32.240760732 +0000 UTC m=+79.000287452" watchObservedRunningTime="2026-07-01 19:22:32.242531519 +0000 UTC m=+79.002058267"
Jul 01 19:22:32.247882 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.247867    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"
Jul 01 19:22:32.258953 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.258928    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg"]
Jul 01 19:22:32.265199 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:32.265175    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbf254d_765f_48a8_941f_febf6e272995.slice/crio-8ee74fda745b159a2881d0aff96f16c87b5ccb70a0ba84a289339ed6e70ef9f8 WatchSource:0}: Error finding container 8ee74fda745b159a2881d0aff96f16c87b5ccb70a0ba84a289339ed6e70ef9f8: Status 404 returned error can't find the container with id 8ee74fda745b159a2881d0aff96f16c87b5ccb70a0ba84a289339ed6e70ef9f8
Jul 01 19:22:32.381899 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:32.381865    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24"]
Jul 01 19:22:32.385221 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:32.385196    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a117e56_ec3c_4923_97b3_a0425bcdae19.slice/crio-4c986ecdef41483cb68efb3eea8ea80ddc433d8f6bd2aad8ff11d990bb6c8ac9 WatchSource:0}: Error finding container 4c986ecdef41483cb68efb3eea8ea80ddc433d8f6bd2aad8ff11d990bb6c8ac9: Status 404 returned error can't find the container with id 4c986ecdef41483cb68efb3eea8ea80ddc433d8f6bd2aad8ff11d990bb6c8ac9
Jul 01 19:22:33.072009 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:33.071974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:33.074969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:33.074940    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:33.204575 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:33.204542    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg" event={"ID":"4dbf254d-765f-48a8-941f-febf6e272995","Type":"ContainerStarted","Data":"8ee74fda745b159a2881d0aff96f16c87b5ccb70a0ba84a289339ed6e70ef9f8"}
Jul 01 19:22:33.205563 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:33.205539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24" event={"ID":"1a117e56-ec3c-4923-97b3-a0425bcdae19","Type":"ContainerStarted","Data":"4c986ecdef41483cb68efb3eea8ea80ddc433d8f6bd2aad8ff11d990bb6c8ac9"}
Jul 01 19:22:33.205748 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:33.205736    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:33.206962 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:33.206945    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75cbd74cc9-tgr8g"
Jul 01 19:22:34.173321 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:34.173294    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ch6qk"
Jul 01 19:22:35.213629 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:35.213599    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24" event={"ID":"1a117e56-ec3c-4923-97b3-a0425bcdae19","Type":"ContainerStarted","Data":"12eecf9c62072b36dd70a6a74d3867843c04a6ba14a93e542d9f938c33ec13f3"}
Jul 01 19:22:35.215436 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:35.215413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6" event={"ID":"f99ef790-7739-42c8-a8fe-5f9545943006","Type":"ContainerStarted","Data":"2c7d35bd773f830aa6f2f1bd263f612c4b609f79127bc4ad5e8bb05287dce451"}
Jul 01 19:22:35.216995 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:35.216976    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg" event={"ID":"4dbf254d-765f-48a8-941f-febf6e272995","Type":"ContainerStarted","Data":"2b94420d79190029f729ecd0d7965ab8d75e3db9bdde6daf5fe1357fbfc42749"}
Jul 01 19:22:35.243599 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:35.243550    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-xcw24" podStartSLOduration=32.604735788 podStartE2EDuration="35.243532126s" totalImagesPullingTime="2.638796338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:35.242506655 +0000 UTC m=+82.002033378" watchObservedRunningTime="2026-07-01 19:22:35.243532126 +0000 UTC m=+82.003058847"
Jul 01 19:22:35.277122 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:35.277072    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-65jxg" podStartSLOduration=33.51433549 podStartE2EDuration="36.277050825s" totalImagesPullingTime="2.762715335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:35.276291654 +0000 UTC m=+82.035818375" watchObservedRunningTime="2026-07-01 19:22:35.277050825 +0000 UTC m=+82.036577547"
Jul 01 19:22:36.222617 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:36.222506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6" event={"ID":"f99ef790-7739-42c8-a8fe-5f9545943006","Type":"ContainerStarted","Data":"da69fe845e13e3a2389e74afd32e8d62b9bea12d7711e8dacecfc0a1c6d6a875"}
Jul 01 19:22:36.251995 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:36.251947    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-v6ns6" podStartSLOduration=33.914065041 podStartE2EDuration="37.251932842s" totalImagesPullingTime="3.337867801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:36.250125167 +0000 UTC m=+83.009651888" watchObservedRunningTime="2026-07-01 19:22:36.251932842 +0000 UTC m=+83.011459563"
Jul 01 19:22:40.157258 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:40.157226    2576 patch_prober.go:28] interesting pod/image-registry-69fb5944f6-lff5w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 01 19:22:40.157639 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:40.157276    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w" podUID="62e6f8b2-4af9-4576-b751-aa4176320cbd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 19:22:40.289631 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:40.289600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:40.291766 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:40.291737    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf0ccf0b-f984-4bcd-8046-8e11b82d19e0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m5nxr\" (UID: \"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0\") " pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:40.562650 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:40.562627    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tz26f\""
Jul 01 19:22:40.570789 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:40.570769    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m5nxr"
Jul 01 19:22:40.694732 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:40.694705    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m5nxr"]
Jul 01 19:22:40.698059 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:40.698031    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0ccf0b_f984_4bcd_8046_8e11b82d19e0.slice/crio-668c9fedc421d696d2be4f90c1cbfaa67f5542909f3ebcd4427d9681d3734d36 WatchSource:0}: Error finding container 668c9fedc421d696d2be4f90c1cbfaa67f5542909f3ebcd4427d9681d3734d36: Status 404 returned error can't find the container with id 668c9fedc421d696d2be4f90c1cbfaa67f5542909f3ebcd4427d9681d3734d36
Jul 01 19:22:41.238875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.238841    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5nxr" event={"ID":"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0","Type":"ContainerStarted","Data":"668c9fedc421d696d2be4f90c1cbfaa67f5542909f3ebcd4427d9681d3734d36"}
Jul 01 19:22:41.833734 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.832325    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-pzprc"]
Jul 01 19:22:41.835981 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.835957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:41.839685 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.838805    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cqdv4\""
Jul 01 19:22:41.840483 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.840100    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 19:22:41.840483 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.840284    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 19:22:41.840483 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.840312    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 19:22:41.851106 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:41.851041    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-pzprc"]
Jul 01 19:22:42.000785 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.000601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b125ad98-8dc8-4196-b516-0246c7b1e8f9-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.000785 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.000681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b125ad98-8dc8-4196-b516-0246c7b1e8f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.000785 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.000749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dlf\" (UniqueName: \"kubernetes.io/projected/b125ad98-8dc8-4196-b516-0246c7b1e8f9-kube-api-access-r5dlf\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.001063 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.000879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b125ad98-8dc8-4196-b516-0246c7b1e8f9-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.102413 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.102285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b125ad98-8dc8-4196-b516-0246c7b1e8f9-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.102413 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.102362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b125ad98-8dc8-4196-b516-0246c7b1e8f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.102649 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.102425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dlf\" (UniqueName: \"kubernetes.io/projected/b125ad98-8dc8-4196-b516-0246c7b1e8f9-kube-api-access-r5dlf\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.102874 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.102756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b125ad98-8dc8-4196-b516-0246c7b1e8f9-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.103102 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.103052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b125ad98-8dc8-4196-b516-0246c7b1e8f9-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.105599 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.105555    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b125ad98-8dc8-4196-b516-0246c7b1e8f9-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.105991 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.105965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b125ad98-8dc8-4196-b516-0246c7b1e8f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.124833 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.124805    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dlf\" (UniqueName: \"kubernetes.io/projected/b125ad98-8dc8-4196-b516-0246c7b1e8f9-kube-api-access-r5dlf\") pod \"prometheus-operator-7b7d896497-pzprc\" (UID: \"b125ad98-8dc8-4196-b516-0246c7b1e8f9\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.152541 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.152514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc"
Jul 01 19:22:42.164048 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.163657    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:22:42.323002 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:42.322922    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-pzprc"]
Jul 01 19:22:42.325779 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:42.325748    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb125ad98_8dc8_4196_b516_0246c7b1e8f9.slice/crio-fd4a3445ac32cc8d999bfa57c6faad41eb2f9cb1a0546fca9b399102c46f948c WatchSource:0}: Error finding container fd4a3445ac32cc8d999bfa57c6faad41eb2f9cb1a0546fca9b399102c46f948c: Status 404 returned error can't find the container with id fd4a3445ac32cc8d999bfa57c6faad41eb2f9cb1a0546fca9b399102c46f948c
Jul 01 19:22:43.250717 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:43.250677    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc" event={"ID":"b125ad98-8dc8-4196-b516-0246c7b1e8f9","Type":"ContainerStarted","Data":"fd4a3445ac32cc8d999bfa57c6faad41eb2f9cb1a0546fca9b399102c46f948c"}
Jul 01 19:22:43.746223 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:43.746196    2576 scope.go:117] "RemoveContainer" containerID="1b3d0fae7adcda87d8b608d7903618dc20aa3ad2765a7c0dc64f44da70876005"
Jul 01 19:22:43.746677 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:43.746388    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7fdbdf9c99-pcrgm_openshift-console-operator(a3b20d7f-4331-4dd8-88ed-bac6347f02e8)\"" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" podUID="a3b20d7f-4331-4dd8-88ed-bac6347f02e8"
Jul 01 19:22:45.261832 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:45.261802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5nxr" event={"ID":"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0","Type":"ContainerStarted","Data":"67cb900cc391baebd3dd1e6847c7afb2013c2dfb87276ca312e03c9db57ee267"}
Jul 01 19:22:45.263494 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:45.263469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc" event={"ID":"b125ad98-8dc8-4196-b516-0246c7b1e8f9","Type":"ContainerStarted","Data":"180c632d66d957a518e9b0b71005d34c2c5e632b2423613f3e6b4cafcb513150"}
Jul 01 19:22:46.268497 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:46.268399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5nxr" event={"ID":"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0","Type":"ContainerStarted","Data":"70dd64e8e4a37d791867d6f21ca2dd91a8d1bc1018ce1bbfefaed0e110c3f716"}
Jul 01 19:22:46.270064 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:46.270034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc" event={"ID":"b125ad98-8dc8-4196-b516-0246c7b1e8f9","Type":"ContainerStarted","Data":"879f01703da9b5bb3def4fc2706600606b75459b3e49de1eabb45586ac313da7"}
Jul 01 19:22:46.289407 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:46.289366    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-pzprc" podStartSLOduration=2.450300932 podStartE2EDuration="5.289354438s" totalImagesPullingTime="2.839053506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:46.288153575 +0000 UTC m=+93.047680329" watchObservedRunningTime="2026-07-01 19:22:46.289354438 +0000 UTC m=+93.048881158"
Jul 01 19:22:48.217383 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.217282    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rcrv8"]
Jul 01 19:22:48.230935 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.230849    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.233309 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.233283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 19:22:48.233682 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.233656    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 19:22:48.233966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.233730    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k8ffm\""
Jul 01 19:22:48.234311 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.234295    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 19:22:48.254151 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254239 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkzp\" (UniqueName: \"kubernetes.io/projected/a1cf8740-ae32-468d-a9ab-12ec40132eec-kube-api-access-8qkzp\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254239 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-textfile\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254340 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1cf8740-ae32-468d-a9ab-12ec40132eec-metrics-client-ca\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254380 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-tls\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254419 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-sys\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254500 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254561 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-root\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.254612 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.254557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-wtmp\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355540 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-sys\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355681 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355681 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-root\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355681 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355594    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-sys\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355681 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-wtmp\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355681 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355668    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-root\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355895 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355895 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355729    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-wtmp\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355895 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkzp\" (UniqueName: \"kubernetes.io/projected/a1cf8740-ae32-468d-a9ab-12ec40132eec-kube-api-access-8qkzp\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355895 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-textfile\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355895 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1cf8740-ae32-468d-a9ab-12ec40132eec-metrics-client-ca\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.355895 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.355830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-tls\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.356160 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:48.355946    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 19:22:48.356160 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:48.356000    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-tls podName:a1cf8740-ae32-468d-a9ab-12ec40132eec nodeName:}" failed. No retries permitted until 2026-07-01 19:22:48.855981898 +0000 UTC m=+95.615508611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-tls") pod "node-exporter-rcrv8" (UID: "a1cf8740-ae32-468d-a9ab-12ec40132eec") : secret "node-exporter-tls" not found
Jul 01 19:22:48.356323 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.356237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-textfile\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.356395 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.356371    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-accelerators-collector-config\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.356507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.356490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1cf8740-ae32-468d-a9ab-12ec40132eec-metrics-client-ca\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.358592 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.358566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.364933 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.364914    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkzp\" (UniqueName: \"kubernetes.io/projected/a1cf8740-ae32-468d-a9ab-12ec40132eec-kube-api-access-8qkzp\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.861821 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.861736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-tls\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:48.864030 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:48.864007    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1cf8740-ae32-468d-a9ab-12ec40132eec-node-exporter-tls\") pod \"node-exporter-rcrv8\" (UID: \"a1cf8740-ae32-468d-a9ab-12ec40132eec\") " pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:49.142255 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.142185    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rcrv8"
Jul 01 19:22:49.152642 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:49.152619    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cf8740_ae32_468d_a9ab_12ec40132eec.slice/crio-f60baac93d4d3f32739fcd00ff94e60beec8cc161a10995621d127f47e247c2b WatchSource:0}: Error finding container f60baac93d4d3f32739fcd00ff94e60beec8cc161a10995621d127f47e247c2b: Status 404 returned error can't find the container with id f60baac93d4d3f32739fcd00ff94e60beec8cc161a10995621d127f47e247c2b
Jul 01 19:22:49.283034 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.283001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m5nxr" event={"ID":"cf0ccf0b-f984-4bcd-8046-8e11b82d19e0","Type":"ContainerStarted","Data":"a141bec120955d3fdf3c896fccee784df95de23ddf0f34dfb32a43723141715b"}
Jul 01 19:22:49.284179 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.284157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcrv8" event={"ID":"a1cf8740-ae32-468d-a9ab-12ec40132eec","Type":"ContainerStarted","Data":"f60baac93d4d3f32739fcd00ff94e60beec8cc161a10995621d127f47e247c2b"}
Jul 01 19:22:49.393123 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.393046    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m5nxr" podStartSLOduration=33.704019549 podStartE2EDuration="41.393032106s" totalImagesPullingTime="7.689012557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:49.368616785 +0000 UTC m=+96.128143505" watchObservedRunningTime="2026-07-01 19:22:49.393032106 +0000 UTC m=+96.152558826"
Jul 01 19:22:49.830223 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.830191    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:22:49.834662 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.834640    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.840232 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.840205    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 19:22:49.840366 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.840347    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 19:22:49.840531 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.840516    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 19:22:49.840610 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.840566    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 19:22:49.841511 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.841490    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k7tb8\""
Jul 01 19:22:49.841607 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.841553    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 19:22:49.841607 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.841574    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 19:22:49.842221 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.842201    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 19:22:49.846913 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.846893    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 19:22:49.847027 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.847000    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 19:22:49.869872 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.869848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870042 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870166 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzb2t\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-kube-api-access-xzb2t\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870283 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870382 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870492 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-web-config\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870603 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870757 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870854 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.870938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.870925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.871023 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.871011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.871123 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.871112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-out\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.871221 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.871209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.888263 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.888239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:22:49.971860 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.971823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972012 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.971871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972012 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.971900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972012 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.971938    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-out\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972012 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.971976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972028    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzb2t\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-kube-api-access-xzb2t\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-web-config\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972664 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:49.972568    2576 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 01 19:22:49.972664 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.972621    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.972664 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:22:49.972641    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls podName:01790832-db80-4cfa-bb1e-cb4d7dd610bf nodeName:}" failed. No retries permitted until 2026-07-01 19:22:50.472621736 +0000 UTC m=+97.232148457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf") : secret "alertmanager-main-tls" not found
Jul 01 19:22:49.973802 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.973423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.973802 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.973503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.976777 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.975931    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-volume\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.976777 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.976515    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-out\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.977508 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.977486    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.977782 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.977756    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.979121 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.978156    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-web-config\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.979121 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.978370    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.979121 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.979009    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:49.979327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:49.979307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:50.006070 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.006045    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzb2t\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-kube-api-access-xzb2t\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:50.149181 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.149154    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q4qsh"
Jul 01 19:22:50.289275 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.289249    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1cf8740-ae32-468d-a9ab-12ec40132eec" containerID="afef14f4c8a40fa1463810744e9689a487d0ebcc21bcb08c51c4c19b43f07e23" exitCode=0
Jul 01 19:22:50.289641 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.289339    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcrv8" event={"ID":"a1cf8740-ae32-468d-a9ab-12ec40132eec","Type":"ContainerDied","Data":"afef14f4c8a40fa1463810744e9689a487d0ebcc21bcb08c51c4c19b43f07e23"}
Jul 01 19:22:50.478280 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.478242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:50.480554 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.480530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:50.746150 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.746075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:22:50.928511 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:50.928431    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:22:50.931025 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:50.930999    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01790832_db80_4cfa_bb1e_cb4d7dd610bf.slice/crio-80dcfa05882e4ccc85e9e52c37cbd0a333b82b7d8a71864e4fd8d7dfdd3f12c4 WatchSource:0}: Error finding container 80dcfa05882e4ccc85e9e52c37cbd0a333b82b7d8a71864e4fd8d7dfdd3f12c4: Status 404 returned error can't find the container with id 80dcfa05882e4ccc85e9e52c37cbd0a333b82b7d8a71864e4fd8d7dfdd3f12c4
Jul 01 19:22:51.295214 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:51.295178    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcrv8" event={"ID":"a1cf8740-ae32-468d-a9ab-12ec40132eec","Type":"ContainerStarted","Data":"96ec36c7a7267bee1e17f3c7d560665396e098e2b3dd0a94311e280990d9fbeb"}
Jul 01 19:22:51.295653 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:51.295231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rcrv8" event={"ID":"a1cf8740-ae32-468d-a9ab-12ec40132eec","Type":"ContainerStarted","Data":"03c6d75c52bc7c115778165fc9902ee2485005ee820da6a32353607bb920309a"}
Jul 01 19:22:51.296427 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:51.296401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerStarted","Data":"80dcfa05882e4ccc85e9e52c37cbd0a333b82b7d8a71864e4fd8d7dfdd3f12c4"}
Jul 01 19:22:51.384992 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:51.384936    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rcrv8" podStartSLOduration=2.393572623 podStartE2EDuration="3.384918638s" totalImagesPullingTime="922.288612ms" totalInitContainerRuntime="69.057403ms" isStatefulPod=true podCreationTimestamp="2026-07-01 19:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:51.384627607 +0000 UTC m=+98.144154328" watchObservedRunningTime="2026-07-01 19:22:51.384918638 +0000 UTC m=+98.144445361"
Jul 01 19:22:52.916052 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.916016    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-557c58445c-mqsq8"]
Jul 01 19:22:52.919193 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.919170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:52.923753 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.923723    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 01 19:22:52.924942 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.924908    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 01 19:22:52.925060 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.925022    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 01 19:22:52.925060 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.925037    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d49aegou6jqim\""
Jul 01 19:22:52.925176 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.925134    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 01 19:22:52.929214 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.929193    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-h6sp8\""
Jul 01 19:22:52.950937 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:52.950916    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-557c58445c-mqsq8"]
Jul 01 19:22:53.002963 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.002936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.003073 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.002974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-secret-metrics-server-client-certs\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.003073 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.002993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-audit-log\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.003073 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.003010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-client-ca-bundle\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.003073 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.003066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwxk\" (UniqueName: \"kubernetes.io/projected/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-kube-api-access-6vwxk\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.003227 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.003083    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-metrics-server-audit-profiles\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.003227 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.003114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-secret-metrics-server-tls\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104552 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-secret-metrics-server-client-certs\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104552 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-audit-log\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104552 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-client-ca-bundle\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104552 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwxk\" (UniqueName: \"kubernetes.io/projected/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-kube-api-access-6vwxk\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104552 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-metrics-server-audit-profiles\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104777 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104704    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-secret-metrics-server-tls\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.104966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.104935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-audit-log\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.105286 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.105262    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.105600 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.105576    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-metrics-server-audit-profiles\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.108279 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.107561    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-client-ca-bundle\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.108279 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.108012    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-secret-metrics-server-client-certs\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.109149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.109128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-secret-metrics-server-tls\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.127753 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.127734    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwxk\" (UniqueName: \"kubernetes.io/projected/766e71e7-dc42-4dd1-bdd7-ff86d19e3804-kube-api-access-6vwxk\") pod \"metrics-server-557c58445c-mqsq8\" (UID: \"766e71e7-dc42-4dd1-bdd7-ff86d19e3804\") " pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.228753 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.228703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:22:53.306386 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.306354    2576 generic.go:356] "Generic (PLEG): container finished" podID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerID="adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e" exitCode=0
Jul 01 19:22:53.306547 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.306506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e"}
Jul 01 19:22:53.419938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.419913    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-557c58445c-mqsq8"]
Jul 01 19:22:53.420136 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:53.420102    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766e71e7_dc42_4dd1_bdd7_ff86d19e3804.slice/crio-c02e8a2f7afd7d49e97afc2de522b19441ce33caaa504fb867a8020d8e30b081 WatchSource:0}: Error finding container c02e8a2f7afd7d49e97afc2de522b19441ce33caaa504fb867a8020d8e30b081: Status 404 returned error can't find the container with id c02e8a2f7afd7d49e97afc2de522b19441ce33caaa504fb867a8020d8e30b081
Jul 01 19:22:53.930222 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:53.930190    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69fb5944f6-lff5w"]
Jul 01 19:22:54.314893 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:54.314823    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8" event={"ID":"766e71e7-dc42-4dd1-bdd7-ff86d19e3804","Type":"ContainerStarted","Data":"c02e8a2f7afd7d49e97afc2de522b19441ce33caaa504fb867a8020d8e30b081"}
Jul 01 19:22:55.745391 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:55.745356    2576 scope.go:117] "RemoveContainer" containerID="1b3d0fae7adcda87d8b608d7903618dc20aa3ad2765a7c0dc64f44da70876005"
Jul 01 19:22:56.324031 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.323998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8" event={"ID":"766e71e7-dc42-4dd1-bdd7-ff86d19e3804","Type":"ContainerStarted","Data":"daf4cf6b4fcc53adcc001437d25edd5e20494905cde445b9ee2be1518f6f6d97"}
Jul 01 19:22:56.327026 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.327000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerStarted","Data":"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"}
Jul 01 19:22:56.327125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.327071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerStarted","Data":"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"}
Jul 01 19:22:56.327125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.327111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerStarted","Data":"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"}
Jul 01 19:22:56.329215 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.329195    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:22:56.329324 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.329273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" event={"ID":"a3b20d7f-4331-4dd8-88ed-bac6347f02e8","Type":"ContainerStarted","Data":"4879ce4931ce2eaa1e60076456a88839cf9484cf2f20f03752b6fd89ba9a9288"}
Jul 01 19:22:56.329577 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.329554    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:22:56.349385 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.349120    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8" podStartSLOduration=1.878527303 podStartE2EDuration="4.349100919s" totalImagesPullingTime="2.470573616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:56.348721322 +0000 UTC m=+103.108248044" watchObservedRunningTime="2026-07-01 19:22:56.349100919 +0000 UTC m=+103.108627638"
Jul 01 19:22:56.373792 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.373749    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm" podStartSLOduration=51.327160595 podStartE2EDuration="57.373729478s" totalImagesPullingTime="6.046568883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:56.373208582 +0000 UTC m=+103.132735328" watchObservedRunningTime="2026-07-01 19:22:56.373729478 +0000 UTC m=+103.133256200"
Jul 01 19:22:56.695786 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.695748    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7fdbdf9c99-pcrgm"
Jul 01 19:22:56.955866 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.955834    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-tq7xb"]
Jul 01 19:22:56.972802 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.972781    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-tq7xb"
Jul 01 19:22:56.978507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.978483    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 19:22:56.978507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.978495    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 19:22:56.979304 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.979283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-784mz\""
Jul 01 19:22:56.981200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:56.981178    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-tq7xb"]
Jul 01 19:22:57.039157 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:57.039122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tshzs\" (UniqueName: \"kubernetes.io/projected/5891a7de-d1fd-4fb1-99cb-473574216100-kube-api-access-tshzs\") pod \"downloads-5499b6fb8f-tq7xb\" (UID: \"5891a7de-d1fd-4fb1-99cb-473574216100\") " pod="openshift-console/downloads-5499b6fb8f-tq7xb"
Jul 01 19:22:57.139739 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:57.139705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tshzs\" (UniqueName: \"kubernetes.io/projected/5891a7de-d1fd-4fb1-99cb-473574216100-kube-api-access-tshzs\") pod \"downloads-5499b6fb8f-tq7xb\" (UID: \"5891a7de-d1fd-4fb1-99cb-473574216100\") " pod="openshift-console/downloads-5499b6fb8f-tq7xb"
Jul 01 19:22:57.151097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:57.151067    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tshzs\" (UniqueName: \"kubernetes.io/projected/5891a7de-d1fd-4fb1-99cb-473574216100-kube-api-access-tshzs\") pod \"downloads-5499b6fb8f-tq7xb\" (UID: \"5891a7de-d1fd-4fb1-99cb-473574216100\") " pod="openshift-console/downloads-5499b6fb8f-tq7xb"
Jul 01 19:22:57.282674 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:57.282650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-tq7xb"
Jul 01 19:22:57.338600 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:57.338561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerStarted","Data":"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"}
Jul 01 19:22:57.338755 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:57.338666    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerStarted","Data":"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"}
Jul 01 19:22:57.563038 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:57.563016    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-tq7xb"]
Jul 01 19:22:57.564923 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:22:57.564902    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5891a7de_d1fd_4fb1_99cb_473574216100.slice/crio-f9623b4fbaf029362579ac3bd01a94929e41ba15e7147fac969cdc2a9c26a6d0 WatchSource:0}: Error finding container f9623b4fbaf029362579ac3bd01a94929e41ba15e7147fac969cdc2a9c26a6d0: Status 404 returned error can't find the container with id f9623b4fbaf029362579ac3bd01a94929e41ba15e7147fac969cdc2a9c26a6d0
Jul 01 19:22:58.347144 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:58.347103    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerStarted","Data":"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"}
Jul 01 19:22:58.348514 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:58.348486    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-tq7xb" event={"ID":"5891a7de-d1fd-4fb1-99cb-473574216100","Type":"ContainerStarted","Data":"f9623b4fbaf029362579ac3bd01a94929e41ba15e7147fac969cdc2a9c26a6d0"}
Jul 01 19:22:58.389306 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:22:58.389255    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.300468589 podStartE2EDuration="9.389237836s" totalImagesPullingTime="5.074479322s" totalInitContainerRuntime="14.289925ms" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:22:58.387638928 +0000 UTC m=+105.147165685" watchObservedRunningTime="2026-07-01 19:22:58.389237836 +0000 UTC m=+105.148764557"
Jul 01 19:23:12.606510 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.606474    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5488b67d56-kqfgp"]
Jul 01 19:23:12.614647 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.614612    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.617463 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.617414    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 01 19:23:12.617463 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.617413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 01 19:23:12.618697 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.618674    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jnpqh\""
Jul 01 19:23:12.618849 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.618727    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 01 19:23:12.619112 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.618985    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 01 19:23:12.619112 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.619002    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 01 19:23:12.620963 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.620941    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5488b67d56-kqfgp"]
Jul 01 19:23:12.623694 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.623673    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 01 19:23:12.782163 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.782123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrc5w\" (UniqueName: \"kubernetes.io/projected/91cf9890-2c89-4a23-ab60-047082256ebe-kube-api-access-vrc5w\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.782444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.782190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-service-ca\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.782444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.782341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-serving-cert\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.782444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.782375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-console-config\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.782444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.782404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-trusted-ca-bundle\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.782671 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.782482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-oauth-config\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.782671 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.782514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-oauth-serving-cert\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.883390 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.883316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-oauth-serving-cert\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.883390 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.883355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrc5w\" (UniqueName: \"kubernetes.io/projected/91cf9890-2c89-4a23-ab60-047082256ebe-kube-api-access-vrc5w\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.883617 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.883493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-service-ca\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.883617 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.883605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-serving-cert\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.883713 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.883637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-console-config\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.883713 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.883668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-trusted-ca-bundle\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.883713 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.883705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-oauth-config\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.884195 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.884147    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-oauth-serving-cert\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.884195 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.884193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-service-ca\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.884388 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.884286    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-console-config\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.884561 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.884541    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-trusted-ca-bundle\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.886418 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.886393    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-oauth-config\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.886614 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.886589    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-serving-cert\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.893331 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.893075    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrc5w\" (UniqueName: \"kubernetes.io/projected/91cf9890-2c89-4a23-ab60-047082256ebe-kube-api-access-vrc5w\") pod \"console-5488b67d56-kqfgp\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") " pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:12.927953 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:12.927919    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:13.229645 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:13.229564    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:23:13.229645 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:13.229634    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:23:13.415324 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:13.415300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5488b67d56-kqfgp"]
Jul 01 19:23:13.417817 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:23:13.417792    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cf9890_2c89_4a23_ab60_047082256ebe.slice/crio-3c5c762260e1d2189fe370b8ebe3dcfac85e7b39379b6bdde17dc265b26ed5f1 WatchSource:0}: Error finding container 3c5c762260e1d2189fe370b8ebe3dcfac85e7b39379b6bdde17dc265b26ed5f1: Status 404 returned error can't find the container with id 3c5c762260e1d2189fe370b8ebe3dcfac85e7b39379b6bdde17dc265b26ed5f1
Jul 01 19:23:14.424136 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:14.424089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-tq7xb" event={"ID":"5891a7de-d1fd-4fb1-99cb-473574216100","Type":"ContainerStarted","Data":"4b24634ea7ca75671b11bd1554e3a8d1e173d59f4cac9ee2508ba1591c8ae5a3"}
Jul 01 19:23:14.425291 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:14.425264    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-tq7xb"
Jul 01 19:23:14.427644 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:14.427611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b67d56-kqfgp" event={"ID":"91cf9890-2c89-4a23-ab60-047082256ebe","Type":"ContainerStarted","Data":"3c5c762260e1d2189fe370b8ebe3dcfac85e7b39379b6bdde17dc265b26ed5f1"}
Jul 01 19:23:14.436885 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:14.436851    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-tq7xb"
Jul 01 19:23:14.458014 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:14.457961    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-tq7xb" podStartSLOduration=2.633002022 podStartE2EDuration="18.457942028s" totalImagesPullingTime="15.824940006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:14.455686197 +0000 UTC m=+121.215212918" watchObservedRunningTime="2026-07-01 19:23:14.457942028 +0000 UTC m=+121.217468750"
Jul 01 19:23:17.444518 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:17.444470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b67d56-kqfgp" event={"ID":"91cf9890-2c89-4a23-ab60-047082256ebe","Type":"ContainerStarted","Data":"5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05"}
Jul 01 19:23:17.466475 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:17.466421    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5488b67d56-kqfgp" podStartSLOduration=1.869074313 podStartE2EDuration="5.466407563s" totalImagesPullingTime="3.59733325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:17.464941966 +0000 UTC m=+124.224468687" watchObservedRunningTime="2026-07-01 19:23:17.466407563 +0000 UTC m=+124.225934283"
Jul 01 19:23:18.450765 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:18.450726    2576 generic.go:356] "Generic (PLEG): container finished" podID="dc491396-37b5-4232-ac52-26beccb0bd39" containerID="0f0116a87ea147b82b1018d32232d6452d694ccd0d609dbf51fe2e9b5b5e54f4" exitCode=0
Jul 01 19:23:18.451246 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:18.450809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-47xvm" event={"ID":"dc491396-37b5-4232-ac52-26beccb0bd39","Type":"ContainerDied","Data":"0f0116a87ea147b82b1018d32232d6452d694ccd0d609dbf51fe2e9b5b5e54f4"}
Jul 01 19:23:18.451364 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:18.451339    2576 scope.go:117] "RemoveContainer" containerID="0f0116a87ea147b82b1018d32232d6452d694ccd0d609dbf51fe2e9b5b5e54f4"
Jul 01 19:23:18.956345 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:18.956294    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w" podUID="62e6f8b2-4af9-4576-b751-aa4176320cbd" containerName="registry" containerID="cri-o://a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b" gracePeriod=30
Jul 01 19:23:19.253836 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.253808    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:23:19.448738 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.448707    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-installation-pull-secrets\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.448968 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.448756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.448968 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.448796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlccw\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-kube-api-access-jlccw\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.449091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.448977    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-trusted-ca\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.449091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.449038    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-image-registry-private-configuration\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.449197 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.449126    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-bound-sa-token\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.449197 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.449166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-certificates\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.449296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.449218    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62e6f8b2-4af9-4576-b751-aa4176320cbd-ca-trust-extracted\") pod \"62e6f8b2-4af9-4576-b751-aa4176320cbd\" (UID: \"62e6f8b2-4af9-4576-b751-aa4176320cbd\") "
Jul 01 19:23:19.449353 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.449335    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:19.449728 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.449524    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-trusted-ca\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.449728 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.449655    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:23:19.453554 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.453406    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:19.453554 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.453437    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:23:19.453985 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.453600    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:23:19.454216 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.454186    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:23:19.454662 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.454633    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-kube-api-access-jlccw" (OuterVolumeSpecName: "kube-api-access-jlccw") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "kube-api-access-jlccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:23:19.458708 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.458132    2576 generic.go:356] "Generic (PLEG): container finished" podID="62e6f8b2-4af9-4576-b751-aa4176320cbd" containerID="a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b" exitCode=0
Jul 01 19:23:19.458708 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.458211    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w"
Jul 01 19:23:19.458708 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.458278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w" event={"ID":"62e6f8b2-4af9-4576-b751-aa4176320cbd","Type":"ContainerDied","Data":"a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b"}
Jul 01 19:23:19.458708 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.458319    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69fb5944f6-lff5w" event={"ID":"62e6f8b2-4af9-4576-b751-aa4176320cbd","Type":"ContainerDied","Data":"565f11038174323847b6516223998219be9931a6237490a5db719ca4463e935a"}
Jul 01 19:23:19.458708 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.458346    2576 scope.go:117] "RemoveContainer" containerID="a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b"
Jul 01 19:23:19.461681 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.461091    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-47xvm" event={"ID":"dc491396-37b5-4232-ac52-26beccb0bd39","Type":"ContainerStarted","Data":"456abf0b97d577a204043f2792095a3778e567e58a5a80607f6ffa940977a127"}
Jul 01 19:23:19.461681 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.461439    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e6f8b2-4af9-4576-b751-aa4176320cbd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "62e6f8b2-4af9-4576-b751-aa4176320cbd" (UID: "62e6f8b2-4af9-4576-b751-aa4176320cbd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:23:19.468597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.468574    2576 scope.go:117] "RemoveContainer" containerID="a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b"
Jul 01 19:23:19.468945 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:23:19.468912    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b\": container with ID starting with a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b not found: ID does not exist" containerID="a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b"
Jul 01 19:23:19.469027 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.468954    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b"} err="failed to get container status \"a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b\": rpc error: code = NotFound desc = could not find container \"a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b\": container with ID starting with a9c07c28abcbfd74505a268f8eb0b9133413833179b55c41429da412d6e28f5b not found: ID does not exist"
Jul 01 19:23:19.550711 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.550672    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62e6f8b2-4af9-4576-b751-aa4176320cbd-ca-trust-extracted\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.550711 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.550703    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-installation-pull-secrets\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.550711 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.550713    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.551004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.550722    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlccw\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-kube-api-access-jlccw\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.551004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.550736    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62e6f8b2-4af9-4576-b751-aa4176320cbd-image-registry-private-configuration\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.551004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.550750    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62e6f8b2-4af9-4576-b751-aa4176320cbd-bound-sa-token\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.551004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.550761    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62e6f8b2-4af9-4576-b751-aa4176320cbd-registry-certificates\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:23:19.777569 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.777542    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69fb5944f6-lff5w"]
Jul 01 19:23:19.781325 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:19.781301    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69fb5944f6-lff5w"]
Jul 01 19:23:21.755734 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:21.755693    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e6f8b2-4af9-4576-b751-aa4176320cbd" path="/var/lib/kubelet/pods/62e6f8b2-4af9-4576-b751-aa4176320cbd/volumes"
Jul 01 19:23:22.577415 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.577375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:23:22.579669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.579644    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a22a2075-444a-40ee-8406-7cd45cd55403-metrics-certs\") pod \"network-metrics-daemon-r2hbh\" (UID: \"a22a2075-444a-40ee-8406-7cd45cd55403\") " pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:23:22.664184 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.664149    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mzk4c\""
Jul 01 19:23:22.672323 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.672301    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r2hbh"
Jul 01 19:23:22.833504 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.833480    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r2hbh"]
Jul 01 19:23:22.835555 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:23:22.835529    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22a2075_444a_40ee_8406_7cd45cd55403.slice/crio-0dd193b59cdc621128787632f600e20a30a875ac17950e1fd4dbffe40ea4a3dc WatchSource:0}: Error finding container 0dd193b59cdc621128787632f600e20a30a875ac17950e1fd4dbffe40ea4a3dc: Status 404 returned error can't find the container with id 0dd193b59cdc621128787632f600e20a30a875ac17950e1fd4dbffe40ea4a3dc
Jul 01 19:23:22.928977 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.928946    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:22.929113 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.929015    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:22.933647 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:22.933626    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:23.481090 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:23.481051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2hbh" event={"ID":"a22a2075-444a-40ee-8406-7cd45cd55403","Type":"ContainerStarted","Data":"0dd193b59cdc621128787632f600e20a30a875ac17950e1fd4dbffe40ea4a3dc"}
Jul 01 19:23:23.484869 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:23.484846    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:23:25.490635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:25.490608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2hbh" event={"ID":"a22a2075-444a-40ee-8406-7cd45cd55403","Type":"ContainerStarted","Data":"84091a4c0f4ad00ce44870e62ad713c088f0f6b84b8129cf2a0106709f3b98ea"}
Jul 01 19:23:26.495799 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:26.495760    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r2hbh" event={"ID":"a22a2075-444a-40ee-8406-7cd45cd55403","Type":"ContainerStarted","Data":"7b9f25a27240851ef30e6f24f80ce98db3f81f0c7fadd7ebefd0b66948e88ebf"}
Jul 01 19:23:26.518811 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:26.518762    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r2hbh" podStartSLOduration=131.055678571 podStartE2EDuration="2m13.518747884s" totalImagesPullingTime="2.463069313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:23:26.517867304 +0000 UTC m=+133.277394027" watchObservedRunningTime="2026-07-01 19:23:26.518747884 +0000 UTC m=+133.278274605"
Jul 01 19:23:32.526074 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:32.526034    2576 generic.go:356] "Generic (PLEG): container finished" podID="740904fe-42d6-4620-8421-048e8ffad910" containerID="958112ce1313c919ff7127e883b76e24cf45e77b5eb7aa0dedf42ff283728464" exitCode=0
Jul 01 19:23:32.526779 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:32.526654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn" event={"ID":"740904fe-42d6-4620-8421-048e8ffad910","Type":"ContainerDied","Data":"958112ce1313c919ff7127e883b76e24cf45e77b5eb7aa0dedf42ff283728464"}
Jul 01 19:23:32.527671 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:32.527653    2576 scope.go:117] "RemoveContainer" containerID="958112ce1313c919ff7127e883b76e24cf45e77b5eb7aa0dedf42ff283728464"
Jul 01 19:23:33.234253 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:33.234222    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:23:33.237945 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:33.237920    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-557c58445c-mqsq8"
Jul 01 19:23:33.531812 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:33.531774    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-c74pn" event={"ID":"740904fe-42d6-4620-8421-048e8ffad910","Type":"ContainerStarted","Data":"4c959551e843b27f5ec2a99c2fc6bbe24df96186891071089924b98f1529f859"}
Jul 01 19:23:37.553987 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:37.553955    2576 generic.go:356] "Generic (PLEG): container finished" podID="d03eca50-fabd-4cd0-9a1c-465c4ee290c0" containerID="577cc16065809468e7c6e46d0c9a14c4c22dc33c08cf04a9d68ab3cf9ab59acd" exitCode=0
Jul 01 19:23:37.554387 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:37.554009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht" event={"ID":"d03eca50-fabd-4cd0-9a1c-465c4ee290c0","Type":"ContainerDied","Data":"577cc16065809468e7c6e46d0c9a14c4c22dc33c08cf04a9d68ab3cf9ab59acd"}
Jul 01 19:23:37.554387 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:37.554320    2576 scope.go:117] "RemoveContainer" containerID="577cc16065809468e7c6e46d0c9a14c4c22dc33c08cf04a9d68ab3cf9ab59acd"
Jul 01 19:23:38.559304 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:23:38.559274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-79964ff6f9-5fhht" event={"ID":"d03eca50-fabd-4cd0-9a1c-465c4ee290c0","Type":"ContainerStarted","Data":"fb412883c4bff48373a66bd6554d268dd77a26e9afbcb38b9c25fee45535e1a8"}
Jul 01 19:24:09.431040 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.430961    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:24:09.431471 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.431255    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="alertmanager" containerID="cri-o://c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd" gracePeriod=120
Jul 01 19:24:09.431471 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.431315    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-metric" containerID="cri-o://1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3" gracePeriod=120
Jul 01 19:24:09.431471 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.431351    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-web" containerID="cri-o://d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f" gracePeriod=120
Jul 01 19:24:09.431471 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.431414    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy" containerID="cri-o://66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9" gracePeriod=120
Jul 01 19:24:09.431471 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.431361    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="prom-label-proxy" containerID="cri-o://97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b" gracePeriod=120
Jul 01 19:24:09.431741 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.431426    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="config-reloader" containerID="cri-o://101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c" gracePeriod=120
Jul 01 19:24:09.684853 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.684780    2576 generic.go:356] "Generic (PLEG): container finished" podID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerID="97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b" exitCode=0
Jul 01 19:24:09.684853 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.684805    2576 generic.go:356] "Generic (PLEG): container finished" podID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerID="66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9" exitCode=0
Jul 01 19:24:09.684853 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.684813    2576 generic.go:356] "Generic (PLEG): container finished" podID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerID="101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c" exitCode=0
Jul 01 19:24:09.684853 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.684821    2576 generic.go:356] "Generic (PLEG): container finished" podID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerID="c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd" exitCode=0
Jul 01 19:24:09.685115 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.684851    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"}
Jul 01 19:24:09.685115 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.684930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"}
Jul 01 19:24:09.685115 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.684973    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"}
Jul 01 19:24:09.685115 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:09.685011    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"}
Jul 01 19:24:10.678966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.678944    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:10.691445 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.691418    2576 generic.go:356] "Generic (PLEG): container finished" podID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerID="1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3" exitCode=0
Jul 01 19:24:10.691546 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.691459    2576 generic.go:356] "Generic (PLEG): container finished" podID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerID="d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f" exitCode=0
Jul 01 19:24:10.691546 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.691499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"}
Jul 01 19:24:10.691546 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.691517    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:10.691643 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.691575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"}
Jul 01 19:24:10.691643 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.691604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"01790832-db80-4cfa-bb1e-cb4d7dd610bf","Type":"ContainerDied","Data":"80dcfa05882e4ccc85e9e52c37cbd0a333b82b7d8a71864e4fd8d7dfdd3f12c4"}
Jul 01 19:24:10.691702 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.691642    2576 scope.go:117] "RemoveContainer" containerID="97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"
Jul 01 19:24:10.698844 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.698828    2576 scope.go:117] "RemoveContainer" containerID="1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"
Jul 01 19:24:10.705788 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.705772    2576 scope.go:117] "RemoveContainer" containerID="66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"
Jul 01 19:24:10.714230 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.714212    2576 scope.go:117] "RemoveContainer" containerID="d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"
Jul 01 19:24:10.722251 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.722233    2576 scope.go:117] "RemoveContainer" containerID="101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"
Jul 01 19:24:10.730771 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.730723    2576 scope.go:117] "RemoveContainer" containerID="c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"
Jul 01 19:24:10.744865 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.744846    2576 scope.go:117] "RemoveContainer" containerID="adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e"
Jul 01 19:24:10.752138 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.752122    2576 scope.go:117] "RemoveContainer" containerID="97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"
Jul 01 19:24:10.752400 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:10.752383    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b\": container with ID starting with 97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b not found: ID does not exist" containerID="97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"
Jul 01 19:24:10.752480 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.752412    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"} err="failed to get container status \"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b\": rpc error: code = NotFound desc = could not find container \"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b\": container with ID starting with 97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b not found: ID does not exist"
Jul 01 19:24:10.752480 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.752427    2576 scope.go:117] "RemoveContainer" containerID="1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"
Jul 01 19:24:10.752663 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:10.752647    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3\": container with ID starting with 1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3 not found: ID does not exist" containerID="1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"
Jul 01 19:24:10.752711 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.752670    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"} err="failed to get container status \"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3\": rpc error: code = NotFound desc = could not find container \"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3\": container with ID starting with 1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3 not found: ID does not exist"
Jul 01 19:24:10.752711 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.752685    2576 scope.go:117] "RemoveContainer" containerID="66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"
Jul 01 19:24:10.752963 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:10.752935    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9\": container with ID starting with 66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9 not found: ID does not exist" containerID="66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"
Jul 01 19:24:10.753069 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.752972    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"} err="failed to get container status \"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9\": rpc error: code = NotFound desc = could not find container \"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9\": container with ID starting with 66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9 not found: ID does not exist"
Jul 01 19:24:10.753069 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.752993    2576 scope.go:117] "RemoveContainer" containerID="d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"
Jul 01 19:24:10.753276 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:10.753230    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f\": container with ID starting with d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f not found: ID does not exist" containerID="d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"
Jul 01 19:24:10.753276 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.753259    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"} err="failed to get container status \"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f\": rpc error: code = NotFound desc = could not find container \"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f\": container with ID starting with d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f not found: ID does not exist"
Jul 01 19:24:10.753408 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.753280    2576 scope.go:117] "RemoveContainer" containerID="101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"
Jul 01 19:24:10.753643 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:10.753624    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c\": container with ID starting with 101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c not found: ID does not exist" containerID="101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"
Jul 01 19:24:10.753740 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.753658    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"} err="failed to get container status \"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c\": rpc error: code = NotFound desc = could not find container \"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c\": container with ID starting with 101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c not found: ID does not exist"
Jul 01 19:24:10.753740 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.753694    2576 scope.go:117] "RemoveContainer" containerID="c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"
Jul 01 19:24:10.753977 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:10.753960    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd\": container with ID starting with c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd not found: ID does not exist" containerID="c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"
Jul 01 19:24:10.754040 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.753982    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"} err="failed to get container status \"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd\": rpc error: code = NotFound desc = could not find container \"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd\": container with ID starting with c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd not found: ID does not exist"
Jul 01 19:24:10.754040 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754000    2576 scope.go:117] "RemoveContainer" containerID="adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e"
Jul 01 19:24:10.754245 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:10.754228    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e\": container with ID starting with adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e not found: ID does not exist" containerID="adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e"
Jul 01 19:24:10.754297 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754250    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e"} err="failed to get container status \"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e\": rpc error: code = NotFound desc = could not find container \"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e\": container with ID starting with adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e not found: ID does not exist"
Jul 01 19:24:10.754297 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754264    2576 scope.go:117] "RemoveContainer" containerID="97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"
Jul 01 19:24:10.754517 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754500    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b"} err="failed to get container status \"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b\": rpc error: code = NotFound desc = could not find container \"97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b\": container with ID starting with 97204c1a517fb2d818ec24dff95aeabefe2174334b02caba57f1183dd046150b not found: ID does not exist"
Jul 01 19:24:10.754568 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754519    2576 scope.go:117] "RemoveContainer" containerID="1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"
Jul 01 19:24:10.754755 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754724    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3"} err="failed to get container status \"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3\": rpc error: code = NotFound desc = could not find container \"1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3\": container with ID starting with 1ac821be63b888eb7c13b122642e012839ef7a73ad5a46b7b4ed01f7c6e3d3c3 not found: ID does not exist"
Jul 01 19:24:10.754805 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754755    2576 scope.go:117] "RemoveContainer" containerID="66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"
Jul 01 19:24:10.755017 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.754998    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9"} err="failed to get container status \"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9\": rpc error: code = NotFound desc = could not find container \"66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9\": container with ID starting with 66b4478a9898c24645a35f99fd240d7bb36ce30ff2ab95b4a5fd77f5b870deb9 not found: ID does not exist"
Jul 01 19:24:10.755088 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755019    2576 scope.go:117] "RemoveContainer" containerID="d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"
Jul 01 19:24:10.755287 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755269    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f"} err="failed to get container status \"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f\": rpc error: code = NotFound desc = could not find container \"d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f\": container with ID starting with d1114a9e781afdccf5fd6dbfde91e56163a8e3f7c9d2eb29da9bfbad54e7a08f not found: ID does not exist"
Jul 01 19:24:10.755287 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755286    2576 scope.go:117] "RemoveContainer" containerID="101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"
Jul 01 19:24:10.755510 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755494    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c"} err="failed to get container status \"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c\": rpc error: code = NotFound desc = could not find container \"101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c\": container with ID starting with 101a9d9256de5972a3b4f173ecfb314d89ae2a419506115dcc6f4cb90cafa75c not found: ID does not exist"
Jul 01 19:24:10.755568 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755511    2576 scope.go:117] "RemoveContainer" containerID="c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"
Jul 01 19:24:10.755724 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755710    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd"} err="failed to get container status \"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd\": rpc error: code = NotFound desc = could not find container \"c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd\": container with ID starting with c0117d00fbb25dd945080013f6544dc21e3286d7f7ca7007ff74915abbe663dd not found: ID does not exist"
Jul 01 19:24:10.755774 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755724    2576 scope.go:117] "RemoveContainer" containerID="adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e"
Jul 01 19:24:10.755944 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.755921    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e"} err="failed to get container status \"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e\": rpc error: code = NotFound desc = could not find container \"adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e\": container with ID starting with adc951de28fa3ac48639e43f56284123c045995e96f070ec0bda035500fbf86e not found: ID does not exist"
Jul 01 19:24:10.850905 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.850883    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-cluster-tls-config\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851000 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.850915    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-out\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851000 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.850942    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-trusted-ca-bundle\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851000 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.850976    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-main-db\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851015    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-tls-assets\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzb2t\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-kube-api-access-xzb2t\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851119    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-metrics-client-ca\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851162    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-web-config\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851202    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851226    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-volume\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851254    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851283    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-web\") pod \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\" (UID: \"01790832-db80-4cfa-bb1e-cb4d7dd610bf\") "
Jul 01 19:24:10.851620 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851387    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:10.851620 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.851437    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:24:10.852151 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.852125    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:10.852311 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.852286    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.852380 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.852318    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-alertmanager-main-db\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.852380 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.852332    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01790832-db80-4cfa-bb1e-cb4d7dd610bf-metrics-client-ca\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.853826 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.853794    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-out" (OuterVolumeSpecName: "config-out") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:24:10.854888 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.854866    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:10.854969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.854886    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:10.855166 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.855105    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-kube-api-access-xzb2t" (OuterVolumeSpecName: "kube-api-access-xzb2t") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "kube-api-access-xzb2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:10.855166 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.855130    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:10.855287 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.855156    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:10.855367 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.855347    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:10.856625 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.856600    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-volume" (OuterVolumeSpecName: "config-volume") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:10.859141 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.859117    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:10.865482 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.865442    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-web-config" (OuterVolumeSpecName: "web-config") pod "01790832-db80-4cfa-bb1e-cb4d7dd610bf" (UID: "01790832-db80-4cfa-bb1e-cb4d7dd610bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:10.953527 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953501    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-web-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953527 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953525    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-main-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953535    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-volume\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953545    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953554    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953566    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-cluster-tls-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953577    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01790832-db80-4cfa-bb1e-cb4d7dd610bf-config-out\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953586    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/01790832-db80-4cfa-bb1e-cb4d7dd610bf-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953597    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-tls-assets\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:10.953667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:10.953605    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzb2t\" (UniqueName: \"kubernetes.io/projected/01790832-db80-4cfa-bb1e-cb4d7dd610bf-kube-api-access-xzb2t\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:11.025181 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.025154    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:24:11.030105 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.030085    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:24:11.081954 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.081931    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:24:11.082201 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082190    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="prom-label-proxy"
Jul 01 19:24:11.082244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082203    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="prom-label-proxy"
Jul 01 19:24:11.082244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082213    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="config-reloader"
Jul 01 19:24:11.082244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082218    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="config-reloader"
Jul 01 19:24:11.082244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082231    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e6f8b2-4af9-4576-b751-aa4176320cbd" containerName="registry"
Jul 01 19:24:11.082244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082236    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e6f8b2-4af9-4576-b751-aa4176320cbd" containerName="registry"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082245    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-metric"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082251    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-metric"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082260    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="alertmanager"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082265    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="alertmanager"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082272    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-web"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082277    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-web"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082286    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="init-config-reloader"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082292    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="init-config-reloader"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082298    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082312    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082356    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-metric"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082364    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy-web"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082370    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="config-reloader"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082376    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="kube-rbac-proxy"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082382    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e6f8b2-4af9-4576-b751-aa4176320cbd" containerName="registry"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082387    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="alertmanager"
Jul 01 19:24:11.082414 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.082393    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" containerName="prom-label-proxy"
Jul 01 19:24:11.087235 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.087219    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.090032 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.090012    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 01 19:24:11.092289 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092267    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 01 19:24:11.092422 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092387    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 01 19:24:11.092551 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 01 19:24:11.092615 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092574    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k7tb8\""
Jul 01 19:24:11.092615 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092584    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 01 19:24:11.092721 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092638    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 01 19:24:11.092721 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092653    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 01 19:24:11.092889 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.092869    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 01 19:24:11.101380 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.101363    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 01 19:24:11.111945 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.111924    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:24:11.255659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvg6\" (UniqueName: \"kubernetes.io/projected/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-kube-api-access-blvg6\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255820 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255820 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255820 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255820 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-web-config\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.255969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255947    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-config-out\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.256169 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.255973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.256169 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.256051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.356993 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.356969    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357092 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357092 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvg6\" (UniqueName: \"kubernetes.io/projected/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-kube-api-access-blvg6\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357092 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357092 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-web-config\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357582 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357582 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357582 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-config-out\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357803 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.357863 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.357839    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.358239 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.358211    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.360260 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.360231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.360354 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.360287    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-config-out\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.360354 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.360335    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.360504 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.360416    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.360565 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.360550    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.360618 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.360597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-web-config\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.360751 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.360730    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.361131 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.361116    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.361539 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.361524    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.366200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.366183    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvg6\" (UniqueName: \"kubernetes.io/projected/68580feb-5129-4e3a-aa78-3c5a8a0eb2d6-kube-api-access-blvg6\") pod \"alertmanager-main-0\" (UID: \"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.396054 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.396035    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 01 19:24:11.540690 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.540657    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 01 19:24:11.543331 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:24:11.543305    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68580feb_5129_4e3a_aa78_3c5a8a0eb2d6.slice/crio-318640e71c1e5582a71987e2f0a5379746d1dbc25ffcf493d79ec41f8986e542 WatchSource:0}: Error finding container 318640e71c1e5582a71987e2f0a5379746d1dbc25ffcf493d79ec41f8986e542: Status 404 returned error can't find the container with id 318640e71c1e5582a71987e2f0a5379746d1dbc25ffcf493d79ec41f8986e542
Jul 01 19:24:11.696669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.696642    2576 generic.go:356] "Generic (PLEG): container finished" podID="68580feb-5129-4e3a-aa78-3c5a8a0eb2d6" containerID="044e22e4f34e0b3a310d47ef163796f4d69a41743d6b313c11886a5774ffb692" exitCode=0
Jul 01 19:24:11.697003 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.696722    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerDied","Data":"044e22e4f34e0b3a310d47ef163796f4d69a41743d6b313c11886a5774ffb692"}
Jul 01 19:24:11.697003 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.696790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerStarted","Data":"318640e71c1e5582a71987e2f0a5379746d1dbc25ffcf493d79ec41f8986e542"}
Jul 01 19:24:11.749576 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:11.749554    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01790832-db80-4cfa-bb1e-cb4d7dd610bf" path="/var/lib/kubelet/pods/01790832-db80-4cfa-bb1e-cb4d7dd610bf/volumes"
Jul 01 19:24:12.704706 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:12.704676    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerStarted","Data":"d423df884a977837c60354e94291b6488be997bdc5c35f1f013896bdd7c6841a"}
Jul 01 19:24:12.705098 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:12.704740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerStarted","Data":"2555be5a6b433606bc52ec94d35b5935e614cfc88400cfe4101531ab0af0f20e"}
Jul 01 19:24:12.705098 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:12.704769    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerStarted","Data":"536ab837277c39e6c449b20457416cccc1eb3c06521193eda73176a606389d5c"}
Jul 01 19:24:12.705098 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:12.704799    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerStarted","Data":"588a058409caa654997b5a77e305b00c11b0eb4867c2e9a67075d2225534a8b8"}
Jul 01 19:24:12.705098 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:12.704827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerStarted","Data":"3ea3b77e8b8e385ced4bed9796aa9f9a28603cbff70e4b737b64c4f79e177cd9"}
Jul 01 19:24:12.705098 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:12.704854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"68580feb-5129-4e3a-aa78-3c5a8a0eb2d6","Type":"ContainerStarted","Data":"62cb968d85bd63559b2ea0699845b11884c0cad8a2299b6b870b76ddf0c7ee4c"}
Jul 01 19:24:12.740752 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:12.740706    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.725446295 podStartE2EDuration="1.740689405s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.24311ms" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:12.738936492 +0000 UTC m=+179.498463250" watchObservedRunningTime="2026-07-01 19:24:12.740689405 +0000 UTC m=+179.500216125"
Jul 01 19:24:13.427463 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.427415    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"]
Jul 01 19:24:13.431426 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.431403    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.434220 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.434194    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 01 19:24:13.434531 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.434194    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 01 19:24:13.434531 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.434267    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 01 19:24:13.434531 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.434296    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 01 19:24:13.434531 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.434267    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 01 19:24:13.435401 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.435384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-brfpz\""
Jul 01 19:24:13.439762 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.439726    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 01 19:24:13.445840 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.445821    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"]
Jul 01 19:24:13.575426 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-telemeter-client-tls\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.575632 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.575632 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-serving-certs-ca-bundle\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.575632 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-metrics-client-ca\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.575632 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqvq\" (UniqueName: \"kubernetes.io/projected/ee4a96eb-1b3d-480e-a0ff-596170396e71-kube-api-access-vvqvq\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.575632 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575599    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-federate-client-tls\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.575878 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-secret-telemeter-client\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.575878 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.575667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676530 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-serving-certs-ca-bundle\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676530 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-metrics-client-ca\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqvq\" (UniqueName: \"kubernetes.io/projected/ee4a96eb-1b3d-480e-a0ff-596170396e71-kube-api-access-vvqvq\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-federate-client-tls\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-secret-telemeter-client\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-telemeter-client-tls\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.676769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.676706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.677295 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.677270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-metrics-client-ca\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.681377 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.681348    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 01 19:24:13.681655 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.681635    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 01 19:24:13.681764 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.681726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 01 19:24:13.681870 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.681855    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 01 19:24:13.681930 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.681916    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 01 19:24:13.687275 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.687250    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 01 19:24:13.687486 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.687382    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-serving-certs-ca-bundle\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.688666 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.688646    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqvq\" (UniqueName: \"kubernetes.io/projected/ee4a96eb-1b3d-480e-a0ff-596170396e71-kube-api-access-vvqvq\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.690285 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.690255    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-secret-telemeter-client\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.690372 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.690299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-federate-client-tls\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.690372 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.690350    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-telemeter-client-tls\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.690372 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.690356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee4a96eb-1b3d-480e-a0ff-596170396e71-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.697807 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.697786    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee4a96eb-1b3d-480e-a0ff-596170396e71-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7988dcbcdb-6hhd4\" (UID: \"ee4a96eb-1b3d-480e-a0ff-596170396e71\") " pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.744882 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.744853    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-brfpz\""
Jul 01 19:24:13.752952 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.752765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"
Jul 01 19:24:13.925621 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:13.925590    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4"]
Jul 01 19:24:13.928981 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:24:13.928922    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4a96eb_1b3d_480e_a0ff_596170396e71.slice/crio-28fda9ab824d1699e45f077eb68c14a9a9a548b85274116cd37157f08a150563 WatchSource:0}: Error finding container 28fda9ab824d1699e45f077eb68c14a9a9a548b85274116cd37157f08a150563: Status 404 returned error can't find the container with id 28fda9ab824d1699e45f077eb68c14a9a9a548b85274116cd37157f08a150563
Jul 01 19:24:14.715802 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:14.715762    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4" event={"ID":"ee4a96eb-1b3d-480e-a0ff-596170396e71","Type":"ContainerStarted","Data":"28fda9ab824d1699e45f077eb68c14a9a9a548b85274116cd37157f08a150563"}
Jul 01 19:24:16.725778 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:16.725742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4" event={"ID":"ee4a96eb-1b3d-480e-a0ff-596170396e71","Type":"ContainerStarted","Data":"fad0454fd70e4a2001916c5efa4fb6ee1ab9625daf0a240a5cfa56e5fd527bbb"}
Jul 01 19:24:16.726192 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:16.725821    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4" event={"ID":"ee4a96eb-1b3d-480e-a0ff-596170396e71","Type":"ContainerStarted","Data":"43f0a7e43e6492285cdf3bff34844e854cbd52df4aa6c263dd272e4e778d5493"}
Jul 01 19:24:16.726192 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:16.725841    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4" event={"ID":"ee4a96eb-1b3d-480e-a0ff-596170396e71","Type":"ContainerStarted","Data":"c60cc6e1f390ce48c6a94dd177869d1037008ce1456d56bd023fa7dcbcbe13f9"}
Jul 01 19:24:16.752836 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:16.752781    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7988dcbcdb-6hhd4" podStartSLOduration=1.644412328 podStartE2EDuration="3.752767115s" totalImagesPullingTime="2.108354787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:16.750231053 +0000 UTC m=+183.509757764" watchObservedRunningTime="2026-07-01 19:24:16.752767115 +0000 UTC m=+183.512293884"
Jul 01 19:24:17.752508 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.752480    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95d54f49c-r9cdv"]
Jul 01 19:24:17.756117 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.756092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.763102 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.763080    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95d54f49c-r9cdv"]
Jul 01 19:24:17.815527 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.815493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-oauth-config\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.815687 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.815536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-config\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.815687 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.815566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-service-ca\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.815687 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.815607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-serving-cert\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.815687 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.815659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-trusted-ca-bundle\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.815926 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.815706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7x8\" (UniqueName: \"kubernetes.io/projected/690a52f8-0b14-4e99-9294-a147ae6b4a7d-kube-api-access-kn7x8\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.815982 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.815918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-oauth-serving-cert\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.917236 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.917199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-trusted-ca-bundle\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.917236 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.917236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7x8\" (UniqueName: \"kubernetes.io/projected/690a52f8-0b14-4e99-9294-a147ae6b4a7d-kube-api-access-kn7x8\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.917482 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.917276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-oauth-serving-cert\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.917482 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.917295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-oauth-config\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.917482 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.917326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-config\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.917482 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.917349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-service-ca\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.917482 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.917378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-serving-cert\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.918100 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.918074    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-service-ca\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.918196 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.918148    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-config\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.918196 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.918153    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-oauth-serving-cert\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.918436 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.918418    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-trusted-ca-bundle\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.920318 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.920291    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-serving-cert\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.920425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.920402    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-oauth-config\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:17.925151 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:17.925124    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7x8\" (UniqueName: \"kubernetes.io/projected/690a52f8-0b14-4e99-9294-a147ae6b4a7d-kube-api-access-kn7x8\") pod \"console-95d54f49c-r9cdv\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") " pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:18.066738 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:18.066704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:18.195099 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:18.195075    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95d54f49c-r9cdv"]
Jul 01 19:24:18.197665 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:24:18.197640    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690a52f8_0b14_4e99_9294_a147ae6b4a7d.slice/crio-558c44e2f4331fb591e1a019e240baf9fbfc5120c950687e2cc4e2d123b60d51 WatchSource:0}: Error finding container 558c44e2f4331fb591e1a019e240baf9fbfc5120c950687e2cc4e2d123b60d51: Status 404 returned error can't find the container with id 558c44e2f4331fb591e1a019e240baf9fbfc5120c950687e2cc4e2d123b60d51
Jul 01 19:24:18.735397 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:18.735359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95d54f49c-r9cdv" event={"ID":"690a52f8-0b14-4e99-9294-a147ae6b4a7d","Type":"ContainerStarted","Data":"5a4eab4ef2560b0b09ec580b5f610dc9bc5519b141c2806231a96b071588714b"}
Jul 01 19:24:18.735594 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:18.735410    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95d54f49c-r9cdv" event={"ID":"690a52f8-0b14-4e99-9294-a147ae6b4a7d","Type":"ContainerStarted","Data":"558c44e2f4331fb591e1a019e240baf9fbfc5120c950687e2cc4e2d123b60d51"}
Jul 01 19:24:18.756130 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:18.756085    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95d54f49c-r9cdv" podStartSLOduration=1.7560686890000001 podStartE2EDuration="1.756068689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:24:18.753783497 +0000 UTC m=+185.513310217" watchObservedRunningTime="2026-07-01 19:24:18.756068689 +0000 UTC m=+185.515595427"
Jul 01 19:24:28.067669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:28.067630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:28.068139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:28.067894    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:28.072609 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:28.072586    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:28.777706 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:28.777680    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:24:28.834599 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:28.834569    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5488b67d56-kqfgp"]
Jul 01 19:24:53.856927 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:53.856888    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5488b67d56-kqfgp" podUID="91cf9890-2c89-4a23-ab60-047082256ebe" containerName="console" containerID="cri-o://5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05" gracePeriod=15
Jul 01 19:24:54.102190 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.102165    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:24:54.189847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.189781    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-service-ca\") pod \"91cf9890-2c89-4a23-ab60-047082256ebe\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") "
Jul 01 19:24:54.189847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.189829    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrc5w\" (UniqueName: \"kubernetes.io/projected/91cf9890-2c89-4a23-ab60-047082256ebe-kube-api-access-vrc5w\") pod \"91cf9890-2c89-4a23-ab60-047082256ebe\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") "
Jul 01 19:24:54.190021 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.189939    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-serving-cert\") pod \"91cf9890-2c89-4a23-ab60-047082256ebe\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") "
Jul 01 19:24:54.190021 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.189970    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-trusted-ca-bundle\") pod \"91cf9890-2c89-4a23-ab60-047082256ebe\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") "
Jul 01 19:24:54.190021 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.189995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-console-config\") pod \"91cf9890-2c89-4a23-ab60-047082256ebe\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") "
Jul 01 19:24:54.190178 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.190049    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-oauth-serving-cert\") pod \"91cf9890-2c89-4a23-ab60-047082256ebe\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") "
Jul 01 19:24:54.190178 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.190097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-oauth-config\") pod \"91cf9890-2c89-4a23-ab60-047082256ebe\" (UID: \"91cf9890-2c89-4a23-ab60-047082256ebe\") "
Jul 01 19:24:54.190178 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.190129    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-service-ca" (OuterVolumeSpecName: "service-ca") pod "91cf9890-2c89-4a23-ab60-047082256ebe" (UID: "91cf9890-2c89-4a23-ab60-047082256ebe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:54.190390 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.190359    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-service-ca\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:54.190390 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.190371    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "91cf9890-2c89-4a23-ab60-047082256ebe" (UID: "91cf9890-2c89-4a23-ab60-047082256ebe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:54.190571 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.190424    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-console-config" (OuterVolumeSpecName: "console-config") pod "91cf9890-2c89-4a23-ab60-047082256ebe" (UID: "91cf9890-2c89-4a23-ab60-047082256ebe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:54.190571 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.190439    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "91cf9890-2c89-4a23-ab60-047082256ebe" (UID: "91cf9890-2c89-4a23-ab60-047082256ebe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:24:54.192052 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.192029    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cf9890-2c89-4a23-ab60-047082256ebe-kube-api-access-vrc5w" (OuterVolumeSpecName: "kube-api-access-vrc5w") pod "91cf9890-2c89-4a23-ab60-047082256ebe" (UID: "91cf9890-2c89-4a23-ab60-047082256ebe"). InnerVolumeSpecName "kube-api-access-vrc5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:24:54.192140 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.192117    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "91cf9890-2c89-4a23-ab60-047082256ebe" (UID: "91cf9890-2c89-4a23-ab60-047082256ebe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:54.192189 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.192142    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "91cf9890-2c89-4a23-ab60-047082256ebe" (UID: "91cf9890-2c89-4a23-ab60-047082256ebe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:24:54.291684 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.291653    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrc5w\" (UniqueName: \"kubernetes.io/projected/91cf9890-2c89-4a23-ab60-047082256ebe-kube-api-access-vrc5w\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:54.291684 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.291678    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-serving-cert\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:54.291684 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.291688    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-trusted-ca-bundle\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:54.291867 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.291698    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-console-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:54.291867 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.291706    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/91cf9890-2c89-4a23-ab60-047082256ebe-oauth-serving-cert\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:54.291867 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.291714    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/91cf9890-2c89-4a23-ab60-047082256ebe-console-oauth-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:24:54.873922 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.873884    2576 generic.go:356] "Generic (PLEG): container finished" podID="91cf9890-2c89-4a23-ab60-047082256ebe" containerID="5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05" exitCode=0
Jul 01 19:24:54.874416 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.873946    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b67d56-kqfgp"
Jul 01 19:24:54.874416 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.873981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b67d56-kqfgp" event={"ID":"91cf9890-2c89-4a23-ab60-047082256ebe","Type":"ContainerDied","Data":"5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05"}
Jul 01 19:24:54.874416 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.874035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b67d56-kqfgp" event={"ID":"91cf9890-2c89-4a23-ab60-047082256ebe","Type":"ContainerDied","Data":"3c5c762260e1d2189fe370b8ebe3dcfac85e7b39379b6bdde17dc265b26ed5f1"}
Jul 01 19:24:54.874416 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.874055    2576 scope.go:117] "RemoveContainer" containerID="5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05"
Jul 01 19:24:54.882844 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.882827    2576 scope.go:117] "RemoveContainer" containerID="5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05"
Jul 01 19:24:54.883086 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:24:54.883068    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05\": container with ID starting with 5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05 not found: ID does not exist" containerID="5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05"
Jul 01 19:24:54.883131 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.883094    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05"} err="failed to get container status \"5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05\": rpc error: code = NotFound desc = could not find container \"5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05\": container with ID starting with 5033730f0e3654847331c0deb29bd3211cd3ce3a74e7ff83d51de78647116e05 not found: ID does not exist"
Jul 01 19:24:54.896063 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.896040    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5488b67d56-kqfgp"]
Jul 01 19:24:54.899535 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:54.899515    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5488b67d56-kqfgp"]
Jul 01 19:24:55.749556 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:24:55.749527    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91cf9890-2c89-4a23-ab60-047082256ebe" path="/var/lib/kubelet/pods/91cf9890-2c89-4a23-ab60-047082256ebe/volumes"
Jul 01 19:25:52.051854 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:25:52.051822    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95d54f49c-r9cdv"]
Jul 01 19:26:13.675495 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:13.675464    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:26:13.680434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:13.680412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:26:13.687031 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:13.687014    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 19:26:17.074025 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.073983    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-95d54f49c-r9cdv" podUID="690a52f8-0b14-4e99-9294-a147ae6b4a7d" containerName="console" containerID="cri-o://5a4eab4ef2560b0b09ec580b5f610dc9bc5519b141c2806231a96b071588714b" gracePeriod=15
Jul 01 19:26:17.191617 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.191591    2576 generic.go:356] "Generic (PLEG): container finished" podID="690a52f8-0b14-4e99-9294-a147ae6b4a7d" containerID="5a4eab4ef2560b0b09ec580b5f610dc9bc5519b141c2806231a96b071588714b" exitCode=0
Jul 01 19:26:17.191718 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.191665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95d54f49c-r9cdv" event={"ID":"690a52f8-0b14-4e99-9294-a147ae6b4a7d","Type":"ContainerDied","Data":"5a4eab4ef2560b0b09ec580b5f610dc9bc5519b141c2806231a96b071588714b"}
Jul 01 19:26:17.319435 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.319412    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:26:17.394982 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.394922    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn7x8\" (UniqueName: \"kubernetes.io/projected/690a52f8-0b14-4e99-9294-a147ae6b4a7d-kube-api-access-kn7x8\") pod \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") "
Jul 01 19:26:17.394982 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.394951    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-oauth-serving-cert\") pod \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") "
Jul 01 19:26:17.395165 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.394990    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-config\") pod \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") "
Jul 01 19:26:17.395165 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395019    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-serving-cert\") pod \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") "
Jul 01 19:26:17.395165 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-service-ca\") pod \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") "
Jul 01 19:26:17.395165 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-oauth-config\") pod \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") "
Jul 01 19:26:17.395367 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-trusted-ca-bundle\") pod \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\" (UID: \"690a52f8-0b14-4e99-9294-a147ae6b4a7d\") "
Jul 01 19:26:17.395428 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395363    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "690a52f8-0b14-4e99-9294-a147ae6b4a7d" (UID: "690a52f8-0b14-4e99-9294-a147ae6b4a7d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:26:17.395517 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395443    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-oauth-serving-cert\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:17.395575 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395504    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-service-ca" (OuterVolumeSpecName: "service-ca") pod "690a52f8-0b14-4e99-9294-a147ae6b4a7d" (UID: "690a52f8-0b14-4e99-9294-a147ae6b4a7d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:26:17.395575 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395508    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-config" (OuterVolumeSpecName: "console-config") pod "690a52f8-0b14-4e99-9294-a147ae6b4a7d" (UID: "690a52f8-0b14-4e99-9294-a147ae6b4a7d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:26:17.395712 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.395589    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "690a52f8-0b14-4e99-9294-a147ae6b4a7d" (UID: "690a52f8-0b14-4e99-9294-a147ae6b4a7d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:26:17.397085 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.397061    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "690a52f8-0b14-4e99-9294-a147ae6b4a7d" (UID: "690a52f8-0b14-4e99-9294-a147ae6b4a7d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:26:17.397174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.397124    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "690a52f8-0b14-4e99-9294-a147ae6b4a7d" (UID: "690a52f8-0b14-4e99-9294-a147ae6b4a7d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:26:17.397174 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.397127    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690a52f8-0b14-4e99-9294-a147ae6b4a7d-kube-api-access-kn7x8" (OuterVolumeSpecName: "kube-api-access-kn7x8") pod "690a52f8-0b14-4e99-9294-a147ae6b4a7d" (UID: "690a52f8-0b14-4e99-9294-a147ae6b4a7d"). InnerVolumeSpecName "kube-api-access-kn7x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:26:17.496499 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.496479    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-oauth-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:17.496499 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.496499    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-trusted-ca-bundle\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:17.496622 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.496508    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn7x8\" (UniqueName: \"kubernetes.io/projected/690a52f8-0b14-4e99-9294-a147ae6b4a7d-kube-api-access-kn7x8\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:17.496622 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.496517    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:17.496622 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.496525    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/690a52f8-0b14-4e99-9294-a147ae6b4a7d-console-serving-cert\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:17.496622 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:17.496533    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/690a52f8-0b14-4e99-9294-a147ae6b4a7d-service-ca\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:18.196257 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:18.196224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95d54f49c-r9cdv" event={"ID":"690a52f8-0b14-4e99-9294-a147ae6b4a7d","Type":"ContainerDied","Data":"558c44e2f4331fb591e1a019e240baf9fbfc5120c950687e2cc4e2d123b60d51"}
Jul 01 19:26:18.196644 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:18.196280    2576 scope.go:117] "RemoveContainer" containerID="5a4eab4ef2560b0b09ec580b5f610dc9bc5519b141c2806231a96b071588714b"
Jul 01 19:26:18.196644 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:18.196234    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95d54f49c-r9cdv"
Jul 01 19:26:18.219519 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:18.219498    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95d54f49c-r9cdv"]
Jul 01 19:26:18.224197 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:18.224174    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-95d54f49c-r9cdv"]
Jul 01 19:26:19.749760 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:19.749718    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="690a52f8-0b14-4e99-9294-a147ae6b4a7d" path="/var/lib/kubelet/pods/690a52f8-0b14-4e99-9294-a147ae6b4a7d/volumes"
Jul 01 19:26:24.260181 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.260137    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"]
Jul 01 19:26:24.260578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.260477    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cf9890-2c89-4a23-ab60-047082256ebe" containerName="console"
Jul 01 19:26:24.260578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.260488    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cf9890-2c89-4a23-ab60-047082256ebe" containerName="console"
Jul 01 19:26:24.260578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.260504    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690a52f8-0b14-4e99-9294-a147ae6b4a7d" containerName="console"
Jul 01 19:26:24.260578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.260510    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="690a52f8-0b14-4e99-9294-a147ae6b4a7d" containerName="console"
Jul 01 19:26:24.260578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.260566    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cf9890-2c89-4a23-ab60-047082256ebe" containerName="console"
Jul 01 19:26:24.260578 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.260577    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="690a52f8-0b14-4e99-9294-a147ae6b4a7d" containerName="console"
Jul 01 19:26:24.264991 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.264972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.267963 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.267943    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 01 19:26:24.267963 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.267958    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-dhmhs\""
Jul 01 19:26:24.269020 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.269006    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 01 19:26:24.273703 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.273681    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"]
Jul 01 19:26:24.344810 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.344777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.344909 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.344823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.344963 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.344911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqtnz\" (UniqueName: \"kubernetes.io/projected/7d4446af-5100-49c9-b8fb-53f35798b168-kube-api-access-hqtnz\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.446122 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.446098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.446224 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.446145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqtnz\" (UniqueName: \"kubernetes.io/projected/7d4446af-5100-49c9-b8fb-53f35798b168-kube-api-access-hqtnz\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.446224 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.446176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.446470 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.446433    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.446537 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.446485    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.457024 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.456999    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqtnz\" (UniqueName: \"kubernetes.io/projected/7d4446af-5100-49c9-b8fb-53f35798b168-kube-api-access-hqtnz\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.573763 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.573742    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:24.707050 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.707030    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"]
Jul 01 19:26:24.708897 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:26:24.708869    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4446af_5100_49c9_b8fb_53f35798b168.slice/crio-c74fd87109bd54d97ad8803198ed01a6498b1e2486d326f57c4ca4ac0fec178b WatchSource:0}: Error finding container c74fd87109bd54d97ad8803198ed01a6498b1e2486d326f57c4ca4ac0fec178b: Status 404 returned error can't find the container with id c74fd87109bd54d97ad8803198ed01a6498b1e2486d326f57c4ca4ac0fec178b
Jul 01 19:26:24.710956 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:24.710937    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 19:26:25.226265 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:25.226229    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2" event={"ID":"7d4446af-5100-49c9-b8fb-53f35798b168","Type":"ContainerStarted","Data":"c74fd87109bd54d97ad8803198ed01a6498b1e2486d326f57c4ca4ac0fec178b"}
Jul 01 19:26:32.258768 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:32.258724    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d4446af-5100-49c9-b8fb-53f35798b168" containerID="507a672d91beeadf585621bd72ccf85ae2c595821c48f6285a3fe9be468dadab" exitCode=0
Jul 01 19:26:32.259133 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:32.258820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2" event={"ID":"7d4446af-5100-49c9-b8fb-53f35798b168","Type":"ContainerDied","Data":"507a672d91beeadf585621bd72ccf85ae2c595821c48f6285a3fe9be468dadab"}
Jul 01 19:26:36.276468 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:36.276420    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d4446af-5100-49c9-b8fb-53f35798b168" containerID="633f37938a16607bca2ed4294628d5608d984ffd2bc7156d7d4c1de617275875" exitCode=0
Jul 01 19:26:36.276926 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:36.276512    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2" event={"ID":"7d4446af-5100-49c9-b8fb-53f35798b168","Type":"ContainerDied","Data":"633f37938a16607bca2ed4294628d5608d984ffd2bc7156d7d4c1de617275875"}
Jul 01 19:26:44.314987 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:44.314953    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d4446af-5100-49c9-b8fb-53f35798b168" containerID="17a68877b6dd73ba32865207653eeb6d6c4ef1af7d8193a83f54d522bd6ba8cb" exitCode=0
Jul 01 19:26:44.315329 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:44.315006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2" event={"ID":"7d4446af-5100-49c9-b8fb-53f35798b168","Type":"ContainerDied","Data":"17a68877b6dd73ba32865207653eeb6d6c4ef1af7d8193a83f54d522bd6ba8cb"}
Jul 01 19:26:45.467873 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.467841    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:45.521593 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.521567    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-util\") pod \"7d4446af-5100-49c9-b8fb-53f35798b168\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") "
Jul 01 19:26:45.521713 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.521635    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-bundle\") pod \"7d4446af-5100-49c9-b8fb-53f35798b168\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") "
Jul 01 19:26:45.521713 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.521680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqtnz\" (UniqueName: \"kubernetes.io/projected/7d4446af-5100-49c9-b8fb-53f35798b168-kube-api-access-hqtnz\") pod \"7d4446af-5100-49c9-b8fb-53f35798b168\" (UID: \"7d4446af-5100-49c9-b8fb-53f35798b168\") "
Jul 01 19:26:45.522251 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.522219    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-bundle" (OuterVolumeSpecName: "bundle") pod "7d4446af-5100-49c9-b8fb-53f35798b168" (UID: "7d4446af-5100-49c9-b8fb-53f35798b168"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:26:45.523840 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.523818    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4446af-5100-49c9-b8fb-53f35798b168-kube-api-access-hqtnz" (OuterVolumeSpecName: "kube-api-access-hqtnz") pod "7d4446af-5100-49c9-b8fb-53f35798b168" (UID: "7d4446af-5100-49c9-b8fb-53f35798b168"). InnerVolumeSpecName "kube-api-access-hqtnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:26:45.526474 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.526438    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-util" (OuterVolumeSpecName: "util") pod "7d4446af-5100-49c9-b8fb-53f35798b168" (UID: "7d4446af-5100-49c9-b8fb-53f35798b168"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 19:26:45.623028 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.622967    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-util\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:45.623028 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.622991    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d4446af-5100-49c9-b8fb-53f35798b168-bundle\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:45.623028 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:45.623005    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqtnz\" (UniqueName: \"kubernetes.io/projected/7d4446af-5100-49c9-b8fb-53f35798b168-kube-api-access-hqtnz\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:26:46.325266 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:46.325226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2" event={"ID":"7d4446af-5100-49c9-b8fb-53f35798b168","Type":"ContainerDied","Data":"c74fd87109bd54d97ad8803198ed01a6498b1e2486d326f57c4ca4ac0fec178b"}
Jul 01 19:26:46.325425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:46.325274    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb58947qm2"
Jul 01 19:26:46.325425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:46.325286    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c74fd87109bd54d97ad8803198ed01a6498b1e2486d326f57c4ca4ac0fec178b"
Jul 01 19:26:52.600993 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.600962    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"]
Jul 01 19:26:52.601327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.601255    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4446af-5100-49c9-b8fb-53f35798b168" containerName="extract"
Jul 01 19:26:52.601327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.601266    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4446af-5100-49c9-b8fb-53f35798b168" containerName="extract"
Jul 01 19:26:52.601327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.601281    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4446af-5100-49c9-b8fb-53f35798b168" containerName="pull"
Jul 01 19:26:52.601327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.601287    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4446af-5100-49c9-b8fb-53f35798b168" containerName="pull"
Jul 01 19:26:52.601327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.601305    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4446af-5100-49c9-b8fb-53f35798b168" containerName="util"
Jul 01 19:26:52.601327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.601310    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4446af-5100-49c9-b8fb-53f35798b168" containerName="util"
Jul 01 19:26:52.601552 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.601360    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4446af-5100-49c9-b8fb-53f35798b168" containerName="extract"
Jul 01 19:26:52.604015 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.604000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:52.608177 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.608150    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 19:26:52.608177 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.608173    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tz8dw\""
Jul 01 19:26:52.608355 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.608179    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 19:26:52.608355 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.608190    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 19:26:52.626122 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.626100    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"]
Jul 01 19:26:52.676125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.676100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2cd\" (UniqueName: \"kubernetes.io/projected/0f4caeef-22b6-4cc1-aa78-e59ce37b19ae-kube-api-access-xl2cd\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-ctr56\" (UID: \"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:52.676223 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.676136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0f4caeef-22b6-4cc1-aa78-e59ce37b19ae-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-ctr56\" (UID: \"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:52.776929 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.776907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0f4caeef-22b6-4cc1-aa78-e59ce37b19ae-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-ctr56\" (UID: \"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:52.777019 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.776967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2cd\" (UniqueName: \"kubernetes.io/projected/0f4caeef-22b6-4cc1-aa78-e59ce37b19ae-kube-api-access-xl2cd\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-ctr56\" (UID: \"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:52.779096 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.779077    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0f4caeef-22b6-4cc1-aa78-e59ce37b19ae-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-ctr56\" (UID: \"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:52.785884 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.785862    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2cd\" (UniqueName: \"kubernetes.io/projected/0f4caeef-22b6-4cc1-aa78-e59ce37b19ae-kube-api-access-xl2cd\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-ctr56\" (UID: \"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:52.913577 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:52.913524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:53.053597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:53.053574    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"]
Jul 01 19:26:53.055786 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:26:53.055755    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4caeef_22b6_4cc1_aa78_e59ce37b19ae.slice/crio-033e915d849fc9138222acd0520bedf9568b57ba4806b406648ed51597216a71 WatchSource:0}: Error finding container 033e915d849fc9138222acd0520bedf9568b57ba4806b406648ed51597216a71: Status 404 returned error can't find the container with id 033e915d849fc9138222acd0520bedf9568b57ba4806b406648ed51597216a71
Jul 01 19:26:53.353544 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:53.353518    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56" event={"ID":"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae","Type":"ContainerStarted","Data":"033e915d849fc9138222acd0520bedf9568b57ba4806b406648ed51597216a71"}
Jul 01 19:26:56.368605 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:56.368516    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56" event={"ID":"0f4caeef-22b6-4cc1-aa78-e59ce37b19ae","Type":"ContainerStarted","Data":"fff536b5755e14827e4dd1f22ef2ced8041f1995ae69c4d1c5b602e63c1339b9"}
Jul 01 19:26:56.368605 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:56.368574    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:26:56.405056 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:56.405012    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56" podStartSLOduration=1.38285259 podStartE2EDuration="4.40499832s" totalImagesPullingTime="3.02214573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:26:56.403566574 +0000 UTC m=+343.163093295" watchObservedRunningTime="2026-07-01 19:26:56.40499832 +0000 UTC m=+343.164525040"
Jul 01 19:26:57.188612 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.188584    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-p9gx4"]
Jul 01 19:26:57.191952 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.191934    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.195737 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.195709    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tdmq4\""
Jul 01 19:26:57.196138 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.196124    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 19:26:57.196316 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.196302    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 01 19:26:57.217308 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.217288    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-p9gx4"]
Jul 01 19:26:57.315911 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.315887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/daa942d6-db34-4635-bd45-43da5edd716e-cabundle0\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.316039 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.315933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7n5\" (UniqueName: \"kubernetes.io/projected/daa942d6-db34-4635-bd45-43da5edd716e-kube-api-access-wx7n5\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.316039 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.316007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/daa942d6-db34-4635-bd45-43da5edd716e-certificates\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.416844 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.416814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/daa942d6-db34-4635-bd45-43da5edd716e-cabundle0\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.417228 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.416871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7n5\" (UniqueName: \"kubernetes.io/projected/daa942d6-db34-4635-bd45-43da5edd716e-kube-api-access-wx7n5\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.417228 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.416914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/daa942d6-db34-4635-bd45-43da5edd716e-certificates\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.417509 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.417490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/daa942d6-db34-4635-bd45-43da5edd716e-cabundle0\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.419209 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.419193    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/daa942d6-db34-4635-bd45-43da5edd716e-certificates\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.434559 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.434538    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7n5\" (UniqueName: \"kubernetes.io/projected/daa942d6-db34-4635-bd45-43da5edd716e-kube-api-access-wx7n5\") pod \"keda-operator-68bff59c-p9gx4\" (UID: \"daa942d6-db34-4635-bd45-43da5edd716e\") " pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.501643 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.501591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:26:57.508346 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.508326    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"]
Jul 01 19:26:57.511692 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.511675    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.515717 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.515694    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 01 19:26:57.541254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.541186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"]
Jul 01 19:26:57.618996 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.618961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-certificates\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.619125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.619004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kt6t\" (UniqueName: \"kubernetes.io/projected/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-kube-api-access-4kt6t\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.619125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.619103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.645573 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.645552    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-p9gx4"]
Jul 01 19:26:57.648027 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:26:57.647993    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa942d6_db34_4635_bd45_43da5edd716e.slice/crio-be001619cedb6dc86f050c1d8ef0f4211cdc10e2805438472b48baf77d03e56c WatchSource:0}: Error finding container be001619cedb6dc86f050c1d8ef0f4211cdc10e2805438472b48baf77d03e56c: Status 404 returned error can't find the container with id be001619cedb6dc86f050c1d8ef0f4211cdc10e2805438472b48baf77d03e56c
Jul 01 19:26:57.720221 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.720195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-certificates\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.720313 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.720233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kt6t\" (UniqueName: \"kubernetes.io/projected/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-kube-api-access-4kt6t\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.720313 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.720283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.720655 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.720626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.722737 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.722718    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-certificates\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.732488 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.732463    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kt6t\" (UniqueName: \"kubernetes.io/projected/cc1fe333-5e6b-4fa1-a4f9-197bd192716f-kube-api-access-4kt6t\") pod \"keda-metrics-apiserver-559f75f947-s2xkv\" (UID: \"cc1fe333-5e6b-4fa1-a4f9-197bd192716f\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.735798 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.735778    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-zrz9d"]
Jul 01 19:26:57.740121 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.740105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:57.743695 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.743679    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 01 19:26:57.752231 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.752164    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-zrz9d"]
Jul 01 19:26:57.821253 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.821223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-certificates\") pod \"keda-admission-5fbd5c4644-zrz9d\" (UID: \"e42f9cf8-27df-420e-ae68-0b81aa78525b\") " pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:57.821399 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.821281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7crt7\" (UniqueName: \"kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-kube-api-access-7crt7\") pod \"keda-admission-5fbd5c4644-zrz9d\" (UID: \"e42f9cf8-27df-420e-ae68-0b81aa78525b\") " pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:57.832173 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.832149    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:26:57.922136 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.922099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-certificates\") pod \"keda-admission-5fbd5c4644-zrz9d\" (UID: \"e42f9cf8-27df-420e-ae68-0b81aa78525b\") " pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:57.922312 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.922159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7crt7\" (UniqueName: \"kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-kube-api-access-7crt7\") pod \"keda-admission-5fbd5c4644-zrz9d\" (UID: \"e42f9cf8-27df-420e-ae68-0b81aa78525b\") " pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:57.922312 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:26:57.922295    2576 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 01 19:26:57.922433 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:26:57.922329    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-zrz9d: secret "keda-admission-webhooks-certs" not found
Jul 01 19:26:57.922433 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:26:57.922399    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-certificates podName:e42f9cf8-27df-420e-ae68-0b81aa78525b nodeName:}" failed. No retries permitted until 2026-07-01 19:26:58.422377776 +0000 UTC m=+345.181904489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-certificates") pod "keda-admission-5fbd5c4644-zrz9d" (UID: "e42f9cf8-27df-420e-ae68-0b81aa78525b") : secret "keda-admission-webhooks-certs" not found
Jul 01 19:26:57.932490 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.932445    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7crt7\" (UniqueName: \"kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-kube-api-access-7crt7\") pod \"keda-admission-5fbd5c4644-zrz9d\" (UID: \"e42f9cf8-27df-420e-ae68-0b81aa78525b\") " pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:57.979721 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:57.979693    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"]
Jul 01 19:26:57.982232 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:26:57.982174    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1fe333_5e6b_4fa1_a4f9_197bd192716f.slice/crio-8a8dd8a41309fd1ee30a0b72e5bf84716a9e6df8ad0d6d6a55c6c8a34741d75d WatchSource:0}: Error finding container 8a8dd8a41309fd1ee30a0b72e5bf84716a9e6df8ad0d6d6a55c6c8a34741d75d: Status 404 returned error can't find the container with id 8a8dd8a41309fd1ee30a0b72e5bf84716a9e6df8ad0d6d6a55c6c8a34741d75d
Jul 01 19:26:58.378978 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:58.378938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv" event={"ID":"cc1fe333-5e6b-4fa1-a4f9-197bd192716f","Type":"ContainerStarted","Data":"8a8dd8a41309fd1ee30a0b72e5bf84716a9e6df8ad0d6d6a55c6c8a34741d75d"}
Jul 01 19:26:58.380144 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:58.380123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-p9gx4" event={"ID":"daa942d6-db34-4635-bd45-43da5edd716e","Type":"ContainerStarted","Data":"be001619cedb6dc86f050c1d8ef0f4211cdc10e2805438472b48baf77d03e56c"}
Jul 01 19:26:58.426614 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:58.426591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-certificates\") pod \"keda-admission-5fbd5c4644-zrz9d\" (UID: \"e42f9cf8-27df-420e-ae68-0b81aa78525b\") " pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:58.429602 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:58.429577    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e42f9cf8-27df-420e-ae68-0b81aa78525b-certificates\") pod \"keda-admission-5fbd5c4644-zrz9d\" (UID: \"e42f9cf8-27df-420e-ae68-0b81aa78525b\") " pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:58.652016 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:58.651938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:26:58.810385 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:58.810354    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-zrz9d"]
Jul 01 19:26:58.812836 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:26:58.812793    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42f9cf8_27df_420e_ae68_0b81aa78525b.slice/crio-72dcf2a453e9ef59d656ad4b7e246d8f34106044a62569bd0944e7784a065098 WatchSource:0}: Error finding container 72dcf2a453e9ef59d656ad4b7e246d8f34106044a62569bd0944e7784a065098: Status 404 returned error can't find the container with id 72dcf2a453e9ef59d656ad4b7e246d8f34106044a62569bd0944e7784a065098
Jul 01 19:26:59.387752 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:26:59.387714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d" event={"ID":"e42f9cf8-27df-420e-ae68-0b81aa78525b","Type":"ContainerStarted","Data":"72dcf2a453e9ef59d656ad4b7e246d8f34106044a62569bd0944e7784a065098"}
Jul 01 19:27:02.403716 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.403684    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv" event={"ID":"cc1fe333-5e6b-4fa1-a4f9-197bd192716f","Type":"ContainerStarted","Data":"c6882af5898f5a6b4fa2265913603b76b9b015aedeb4db46ff03d6a9c8b46c61"}
Jul 01 19:27:02.404066 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.403751    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:27:02.405520 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.405489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-p9gx4" event={"ID":"daa942d6-db34-4635-bd45-43da5edd716e","Type":"ContainerStarted","Data":"04e9bf78bf240f981f2e79c60e76632d5d29929faa3e6d711f4d4daad90db249"}
Jul 01 19:27:02.405647 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.405580    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:27:02.407140 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.407121    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d" event={"ID":"e42f9cf8-27df-420e-ae68-0b81aa78525b","Type":"ContainerStarted","Data":"2b82bd07334ce751337d1c0c8f6846b5b5d3e879df0f240582670b0881d886a4"}
Jul 01 19:27:02.407253 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.407235    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:27:02.426618 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.426323    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv" podStartSLOduration=1.092835497 podStartE2EDuration="5.426305523s" totalImagesPullingTime="4.333470026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:02.425255079 +0000 UTC m=+349.184781802" watchObservedRunningTime="2026-07-01 19:27:02.426305523 +0000 UTC m=+349.185832245"
Jul 01 19:27:02.450228 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.450165    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-p9gx4" podStartSLOduration=0.785582761 podStartE2EDuration="5.450143766s" totalImagesPullingTime="4.664561005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:02.449182568 +0000 UTC m=+349.208709290" watchObservedRunningTime="2026-07-01 19:27:02.450143766 +0000 UTC m=+349.209670489"
Jul 01 19:27:02.470201 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:02.470132    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d" podStartSLOduration=1.967269111 podStartE2EDuration="5.470116775s" totalImagesPullingTime="3.502847664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:27:02.467994791 +0000 UTC m=+349.227521533" watchObservedRunningTime="2026-07-01 19:27:02.470116775 +0000 UTC m=+349.229643496"
Jul 01 19:27:13.415534 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:13.415437    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-s2xkv"
Jul 01 19:27:17.375642 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:17.375605    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-ctr56"
Jul 01 19:27:23.413533 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:23.413503    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-p9gx4"
Jul 01 19:27:23.414018 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:27:23.413758    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-zrz9d"
Jul 01 19:28:05.175543 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.175504    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-sj2m2"]
Jul 01 19:28:05.178837 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.178813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.181929 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.181908    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 19:28:05.182041 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.181943    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 19:28:05.183003 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.182988    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 19:28:05.183124 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.183107    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-b84r8\""
Jul 01 19:28:05.191534 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.191514    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sj2m2"]
Jul 01 19:28:05.232646 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.232624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmq8s\" (UniqueName: \"kubernetes.io/projected/c693859a-f156-4a43-b3f7-4ccbfdaae776-kube-api-access-pmq8s\") pod \"seaweedfs-64568bcd49-sj2m2\" (UID: \"c693859a-f156-4a43-b3f7-4ccbfdaae776\") " pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.232736 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.232671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c693859a-f156-4a43-b3f7-4ccbfdaae776-data\") pod \"seaweedfs-64568bcd49-sj2m2\" (UID: \"c693859a-f156-4a43-b3f7-4ccbfdaae776\") " pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.333723 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.333703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c693859a-f156-4a43-b3f7-4ccbfdaae776-data\") pod \"seaweedfs-64568bcd49-sj2m2\" (UID: \"c693859a-f156-4a43-b3f7-4ccbfdaae776\") " pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.333814 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.333766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmq8s\" (UniqueName: \"kubernetes.io/projected/c693859a-f156-4a43-b3f7-4ccbfdaae776-kube-api-access-pmq8s\") pod \"seaweedfs-64568bcd49-sj2m2\" (UID: \"c693859a-f156-4a43-b3f7-4ccbfdaae776\") " pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.334074 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.334056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c693859a-f156-4a43-b3f7-4ccbfdaae776-data\") pod \"seaweedfs-64568bcd49-sj2m2\" (UID: \"c693859a-f156-4a43-b3f7-4ccbfdaae776\") " pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.345084 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.345058    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmq8s\" (UniqueName: \"kubernetes.io/projected/c693859a-f156-4a43-b3f7-4ccbfdaae776-kube-api-access-pmq8s\") pod \"seaweedfs-64568bcd49-sj2m2\" (UID: \"c693859a-f156-4a43-b3f7-4ccbfdaae776\") " pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.488846 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.488797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:05.636919 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.636888    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-sj2m2"]
Jul 01 19:28:05.639922 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:28:05.639893    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc693859a_f156_4a43_b3f7_4ccbfdaae776.slice/crio-df0466f72cc946a95b0330c90572a465f53aff72cc782b9e883414187e0a4d77 WatchSource:0}: Error finding container df0466f72cc946a95b0330c90572a465f53aff72cc782b9e883414187e0a4d77: Status 404 returned error can't find the container with id df0466f72cc946a95b0330c90572a465f53aff72cc782b9e883414187e0a4d77
Jul 01 19:28:05.680008 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:05.679980    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sj2m2" event={"ID":"c693859a-f156-4a43-b3f7-4ccbfdaae776","Type":"ContainerStarted","Data":"df0466f72cc946a95b0330c90572a465f53aff72cc782b9e883414187e0a4d77"}
Jul 01 19:28:09.700926 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:09.700891    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-sj2m2" event={"ID":"c693859a-f156-4a43-b3f7-4ccbfdaae776","Type":"ContainerStarted","Data":"cec4c36070feb25b4b21a7deb3ee21bb1806e42d59624abe21952f8e74adadc0"}
Jul 01 19:28:09.701383 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:09.701005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:28:09.725865 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:09.725812    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-sj2m2" podStartSLOduration=1.151138609 podStartE2EDuration="4.725797095s" totalImagesPullingTime="3.574658486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:28:09.722618926 +0000 UTC m=+416.482145647" watchObservedRunningTime="2026-07-01 19:28:09.725797095 +0000 UTC m=+416.485323815"
Jul 01 19:28:15.707197 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:28:15.707165    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-sj2m2"
Jul 01 19:29:15.736685 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.736648    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-frxzh"]
Jul 01 19:29:15.740063 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.740043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:15.747550 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.747530    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z9k8w\""
Jul 01 19:29:15.747686 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.747666    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 01 19:29:15.751932 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.751909    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cvgz7"]
Jul 01 19:29:15.755383 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.755365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:15.755995 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.755977    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-frxzh"]
Jul 01 19:29:15.758584 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.758565    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 19:29:15.759302 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.759282    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-smtxn\""
Jul 01 19:29:15.774837 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.774815    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cvgz7"]
Jul 01 19:29:15.790411 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.790384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16af8517-1253-40fa-8ac5-65130ddf87d0-tls-certs\") pod \"model-serving-api-fd65d7d6-frxzh\" (UID: \"16af8517-1253-40fa-8ac5-65130ddf87d0\") " pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:15.790531 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.790513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc47z\" (UniqueName: \"kubernetes.io/projected/16af8517-1253-40fa-8ac5-65130ddf87d0-kube-api-access-pc47z\") pod \"model-serving-api-fd65d7d6-frxzh\" (UID: \"16af8517-1253-40fa-8ac5-65130ddf87d0\") " pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:15.891635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.891613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc47z\" (UniqueName: \"kubernetes.io/projected/16af8517-1253-40fa-8ac5-65130ddf87d0-kube-api-access-pc47z\") pod \"model-serving-api-fd65d7d6-frxzh\" (UID: \"16af8517-1253-40fa-8ac5-65130ddf87d0\") " pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:15.891747 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.891663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16af8517-1253-40fa-8ac5-65130ddf87d0-tls-certs\") pod \"model-serving-api-fd65d7d6-frxzh\" (UID: \"16af8517-1253-40fa-8ac5-65130ddf87d0\") " pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:15.891747 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.891689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f96d07-dc86-4757-9bc4-556fb2a00d66-cert\") pod \"odh-model-controller-84b54b7f97-cvgz7\" (UID: \"74f96d07-dc86-4757-9bc4-556fb2a00d66\") " pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:15.891747 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.891704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrxb\" (UniqueName: \"kubernetes.io/projected/74f96d07-dc86-4757-9bc4-556fb2a00d66-kube-api-access-lwrxb\") pod \"odh-model-controller-84b54b7f97-cvgz7\" (UID: \"74f96d07-dc86-4757-9bc4-556fb2a00d66\") " pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:15.894021 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.894000    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16af8517-1253-40fa-8ac5-65130ddf87d0-tls-certs\") pod \"model-serving-api-fd65d7d6-frxzh\" (UID: \"16af8517-1253-40fa-8ac5-65130ddf87d0\") " pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:15.900146 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.900122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc47z\" (UniqueName: \"kubernetes.io/projected/16af8517-1253-40fa-8ac5-65130ddf87d0-kube-api-access-pc47z\") pod \"model-serving-api-fd65d7d6-frxzh\" (UID: \"16af8517-1253-40fa-8ac5-65130ddf87d0\") " pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:15.993059 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.992986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f96d07-dc86-4757-9bc4-556fb2a00d66-cert\") pod \"odh-model-controller-84b54b7f97-cvgz7\" (UID: \"74f96d07-dc86-4757-9bc4-556fb2a00d66\") " pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:15.993059 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.993017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrxb\" (UniqueName: \"kubernetes.io/projected/74f96d07-dc86-4757-9bc4-556fb2a00d66-kube-api-access-lwrxb\") pod \"odh-model-controller-84b54b7f97-cvgz7\" (UID: \"74f96d07-dc86-4757-9bc4-556fb2a00d66\") " pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:15.995387 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:15.995368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f96d07-dc86-4757-9bc4-556fb2a00d66-cert\") pod \"odh-model-controller-84b54b7f97-cvgz7\" (UID: \"74f96d07-dc86-4757-9bc4-556fb2a00d66\") " pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:16.000911 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:16.000890    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrxb\" (UniqueName: \"kubernetes.io/projected/74f96d07-dc86-4757-9bc4-556fb2a00d66-kube-api-access-lwrxb\") pod \"odh-model-controller-84b54b7f97-cvgz7\" (UID: \"74f96d07-dc86-4757-9bc4-556fb2a00d66\") " pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:16.050723 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:16.050702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:16.065560 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:16.065540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:16.185718 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:16.185688    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-frxzh"]
Jul 01 19:29:16.189308 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:29:16.189279    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16af8517_1253_40fa_8ac5_65130ddf87d0.slice/crio-ab246409d12bdf3f3016bbf3a5307fd141717c5d51f52d22aa662410004c37e4 WatchSource:0}: Error finding container ab246409d12bdf3f3016bbf3a5307fd141717c5d51f52d22aa662410004c37e4: Status 404 returned error can't find the container with id ab246409d12bdf3f3016bbf3a5307fd141717c5d51f52d22aa662410004c37e4
Jul 01 19:29:16.210353 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:16.210292    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-cvgz7"]
Jul 01 19:29:16.212656 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:29:16.212635    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f96d07_dc86_4757_9bc4_556fb2a00d66.slice/crio-d918b52f2da127a08ed00f50b739aa29250f13d8a730f05b4083d19bbb30f0de WatchSource:0}: Error finding container d918b52f2da127a08ed00f50b739aa29250f13d8a730f05b4083d19bbb30f0de: Status 404 returned error can't find the container with id d918b52f2da127a08ed00f50b739aa29250f13d8a730f05b4083d19bbb30f0de
Jul 01 19:29:16.999762 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:16.999723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-cvgz7" event={"ID":"74f96d07-dc86-4757-9bc4-556fb2a00d66","Type":"ContainerStarted","Data":"d918b52f2da127a08ed00f50b739aa29250f13d8a730f05b4083d19bbb30f0de"}
Jul 01 19:29:17.002489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:17.002444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-frxzh" event={"ID":"16af8517-1253-40fa-8ac5-65130ddf87d0","Type":"ContainerStarted","Data":"ab246409d12bdf3f3016bbf3a5307fd141717c5d51f52d22aa662410004c37e4"}
Jul 01 19:29:21.024898 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:21.024866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-frxzh" event={"ID":"16af8517-1253-40fa-8ac5-65130ddf87d0","Type":"ContainerStarted","Data":"afcd9375fb4be2b414f58bb1572c804c9d3c07bd054df366543d07c2485e500b"}
Jul 01 19:29:21.025354 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:21.024943    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:29:21.026339 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:21.026321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-cvgz7" event={"ID":"74f96d07-dc86-4757-9bc4-556fb2a00d66","Type":"ContainerStarted","Data":"1e3a631bbe3be54a0d40fd65e324b938f12d78ec156a8506677266eee9149f2d"}
Jul 01 19:29:21.026436 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:21.026424    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:21.071001 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:21.070950    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-frxzh" podStartSLOduration=2.099049246 podStartE2EDuration="6.0709378s" totalImagesPullingTime="3.971888554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:29:21.051066791 +0000 UTC m=+487.810593529" watchObservedRunningTime="2026-07-01 19:29:21.0709378 +0000 UTC m=+487.830464520"
Jul 01 19:29:21.071154 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:21.071035    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-cvgz7" podStartSLOduration=2.115604942 podStartE2EDuration="6.071030843s" totalImagesPullingTime="3.955425901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:29:21.069372572 +0000 UTC m=+487.828899296" watchObservedRunningTime="2026-07-01 19:29:21.071030843 +0000 UTC m=+487.830557564"
Jul 01 19:29:32.032802 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:32.032767    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-cvgz7"
Jul 01 19:29:32.034625 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:29:32.034601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-frxzh"
Jul 01 19:31:03.862788 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.862753    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"]
Jul 01 19:31:03.866425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.866404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:03.869048 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.869022    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-79b53-predictor-serving-cert\""
Jul 01 19:31:03.869166 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.869144    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xs8vf\""
Jul 01 19:31:03.869166 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.869153    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 19:31:03.870227 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.870206    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-79b53-kube-rbac-proxy-sar-config\""
Jul 01 19:31:03.870413 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.870392    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 19:31:03.877298 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:03.877276    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"]
Jul 01 19:31:04.017126 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.017097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75bc0071-6882-493b-80ab-4ac0b0c2f01c-proxy-tls\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.017296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.017150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwpd\" (UniqueName: \"kubernetes.io/projected/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-api-access-lnwpd\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.017296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.017243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.065783 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.065750    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"]
Jul 01 19:31:04.069242 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.069225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.072072 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.072045    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-79b53-predictor-serving-cert\""
Jul 01 19:31:04.072176 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.072093    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-79b53-kube-rbac-proxy-sar-config\""
Jul 01 19:31:04.079052 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.079028    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"]
Jul 01 19:31:04.118132 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.118064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75bc0071-6882-493b-80ab-4ac0b0c2f01c-proxy-tls\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.118243 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.118140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwpd\" (UniqueName: \"kubernetes.io/projected/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-api-access-lnwpd\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.118243 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.118203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.119004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.118980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.120501 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.120481    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75bc0071-6882-493b-80ab-4ac0b0c2f01c-proxy-tls\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.127631 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.127611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwpd\" (UniqueName: \"kubernetes.io/projected/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-api-access-lnwpd\") pod \"success-200-isvc-79b53-predictor-64ff7f459d-xjf2s\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") " pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.179893 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.179870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:04.218975 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.218811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rkw\" (UniqueName: \"kubernetes.io/projected/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-api-access-64rkw\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.218975 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.218866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-proxy-tls\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.218975 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.218932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.312981 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.312956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"]
Jul 01 19:31:04.315092 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:31:04.315063    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bc0071_6882_493b_80ab_4ac0b0c2f01c.slice/crio-6817df1978b1979ff7284ae6737779445f5ff5c694075d0674f2b35f85f99792 WatchSource:0}: Error finding container 6817df1978b1979ff7284ae6737779445f5ff5c694075d0674f2b35f85f99792: Status 404 returned error can't find the container with id 6817df1978b1979ff7284ae6737779445f5ff5c694075d0674f2b35f85f99792
Jul 01 19:31:04.319596 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.319576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rkw\" (UniqueName: \"kubernetes.io/projected/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-api-access-64rkw\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.319683 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.319609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-proxy-tls\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.319683 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.319645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.320378 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.320357    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.322263 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.322238    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-proxy-tls\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.328231 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.328207    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rkw\" (UniqueName: \"kubernetes.io/projected/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-api-access-64rkw\") pod \"error-404-isvc-79b53-predictor-58774dd6b7-tzcmk\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") " pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.380393 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.380347    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:04.504573 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.504531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" event={"ID":"75bc0071-6882-493b-80ab-4ac0b0c2f01c","Type":"ContainerStarted","Data":"6817df1978b1979ff7284ae6737779445f5ff5c694075d0674f2b35f85f99792"}
Jul 01 19:31:04.516974 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:04.516952    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"]
Jul 01 19:31:04.518235 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:31:04.518215    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f04157_2a0a_4c48_93e8_7c17c1ee0b8a.slice/crio-ca6d6cd329fb3a496e805963937fc35b67796842fdff38a4ffec5f1dcb29acaa WatchSource:0}: Error finding container ca6d6cd329fb3a496e805963937fc35b67796842fdff38a4ffec5f1dcb29acaa: Status 404 returned error can't find the container with id ca6d6cd329fb3a496e805963937fc35b67796842fdff38a4ffec5f1dcb29acaa
Jul 01 19:31:05.529817 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:05.529774    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" event={"ID":"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a","Type":"ContainerStarted","Data":"ca6d6cd329fb3a496e805963937fc35b67796842fdff38a4ffec5f1dcb29acaa"}
Jul 01 19:31:20.334405 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:20.334373    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:31:20.336709 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:20.335278    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:31:20.633499 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:20.633401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" event={"ID":"75bc0071-6882-493b-80ab-4ac0b0c2f01c","Type":"ContainerStarted","Data":"fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9"}
Jul 01 19:31:20.634932 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:20.634904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" event={"ID":"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a","Type":"ContainerStarted","Data":"dd8f0a9d1c7c9c03756cac1af1c8b4c9ab6b2fd7a822b703883749edfd6e8c28"}
Jul 01 19:31:23.663903 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.663865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" event={"ID":"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a","Type":"ContainerStarted","Data":"6d3c678d342109c5e983560e61a6f21df364dfd0b980f7035c3bbba2c5b75f4d"}
Jul 01 19:31:23.664357 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.664100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:23.665867 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.665833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" event={"ID":"75bc0071-6882-493b-80ab-4ac0b0c2f01c","Type":"ContainerStarted","Data":"25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735"}
Jul 01 19:31:23.666030 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.666018    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:23.666083 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.666045    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:23.667490 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.667461    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 19:31:23.686980 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.686927    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podStartSLOduration=1.256832121 podStartE2EDuration="19.686890424s" totalImagesPullingTime="18.430058303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:31:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:31:23.683844551 +0000 UTC m=+610.443371273" watchObservedRunningTime="2026-07-01 19:31:23.686890424 +0000 UTC m=+610.446417148"
Jul 01 19:31:23.713864 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:23.713828    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podStartSLOduration=2.072919806 podStartE2EDuration="20.713816066s" totalImagesPullingTime="18.64089626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:31:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:31:23.71370171 +0000 UTC m=+610.473228458" watchObservedRunningTime="2026-07-01 19:31:23.713816066 +0000 UTC m=+610.473342787"
Jul 01 19:31:24.670606 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:24.670565    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 19:31:24.671067 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:24.670686    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:24.671854 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:24.671834    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 19:31:25.674989 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:25.674944    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 19:31:29.675732 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:29.675701    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:31:29.676300 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:29.676268    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 19:31:30.679804 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:30.679776    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:31:30.680438 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:30.680410    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 19:31:39.676866 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:39.676781    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 19:31:40.680999 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:40.680963    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 19:31:49.677115 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:49.677070    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 19:31:50.680616 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:50.680578    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 19:31:59.677172 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:31:59.677130    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 01 19:32:00.680824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:00.680784    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 01 19:32:09.677354 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:09.677325    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:32:10.681645 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:10.681613    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:32:33.805297 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.805264    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"]
Jul 01 19:32:33.805716 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.805492    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container" containerID="cri-o://dd8f0a9d1c7c9c03756cac1af1c8b4c9ab6b2fd7a822b703883749edfd6e8c28" gracePeriod=30
Jul 01 19:32:33.805716 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.805536    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kube-rbac-proxy" containerID="cri-o://6d3c678d342109c5e983560e61a6f21df364dfd0b980f7035c3bbba2c5b75f4d" gracePeriod=30
Jul 01 19:32:33.871919 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.871885    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"]
Jul 01 19:32:33.872103 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.872082    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container" containerID="cri-o://fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9" gracePeriod=30
Jul 01 19:32:33.872180 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.872115    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kube-rbac-proxy" containerID="cri-o://25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735" gracePeriod=30
Jul 01 19:32:33.878944 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.878919    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"]
Jul 01 19:32:33.882824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.882803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:33.885435 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.885393    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6c6c1-predictor-serving-cert\""
Jul 01 19:32:33.885718 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.885702    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6c6c1-kube-rbac-proxy-sar-config\""
Jul 01 19:32:33.891718 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.891685    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"]
Jul 01 19:32:33.960788 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.960755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:33.960938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.960803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpw89\" (UniqueName: \"kubernetes.io/projected/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-api-access-fpw89\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:33.960938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.960852    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:33.980092 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.980068    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"]
Jul 01 19:32:33.983636 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.983615    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:33.983751 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.983704    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerID="6d3c678d342109c5e983560e61a6f21df364dfd0b980f7035c3bbba2c5b75f4d" exitCode=2
Jul 01 19:32:33.983816 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.983790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" event={"ID":"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a","Type":"ContainerDied","Data":"6d3c678d342109c5e983560e61a6f21df364dfd0b980f7035c3bbba2c5b75f4d"}
Jul 01 19:32:33.986204 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.986181    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6c6c1-kube-rbac-proxy-sar-config\""
Jul 01 19:32:33.986336 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.986185    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-6c6c1-predictor-serving-cert\""
Jul 01 19:32:33.996199 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:33.996174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"]
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.062557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.062654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw89\" (UniqueName: \"kubernetes.io/projected/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-api-access-fpw89\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.062701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ad769a-93dd-4cf1-b3c9-536391a9732c-proxy-tls\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.062771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.062807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.062865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmzw\" (UniqueName: \"kubernetes.io/projected/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-api-access-jkmzw\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:32:34.063031    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-serving-cert: secret "success-200-isvc-6c6c1-predictor-serving-cert" not found
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:32:34.063172    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls podName:00028cdb-adec-4d60-a662-dd3dfa6fba7e nodeName:}" failed. No retries permitted until 2026-07-01 19:32:34.563141703 +0000 UTC m=+681.322668404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls") pod "success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" (UID: "00028cdb-adec-4d60-a662-dd3dfa6fba7e") : secret "success-200-isvc-6c6c1-predictor-serving-cert" not found
Jul 01 19:32:34.068091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.064206    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.071710 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.071686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw89\" (UniqueName: \"kubernetes.io/projected/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-api-access-fpw89\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.163884 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.163849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.164016 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.163898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmzw\" (UniqueName: \"kubernetes.io/projected/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-api-access-jkmzw\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.164016 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.163954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ad769a-93dd-4cf1-b3c9-536391a9732c-proxy-tls\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.164534 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.164513    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.166138 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.166122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ad769a-93dd-4cf1-b3c9-536391a9732c-proxy-tls\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.172579 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.172560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmzw\" (UniqueName: \"kubernetes.io/projected/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-api-access-jkmzw\") pod \"error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") " pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.293639 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.293618    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:34.424833 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.424809    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"]
Jul 01 19:32:34.426751 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:32:34.426722    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ad769a_93dd_4cf1_b3c9_536391a9732c.slice/crio-c415629e47fc1bd773d392c959ea7773edc51dd4f91652ceadfb8dd0f3269f9c WatchSource:0}: Error finding container c415629e47fc1bd773d392c959ea7773edc51dd4f91652ceadfb8dd0f3269f9c: Status 404 returned error can't find the container with id c415629e47fc1bd773d392c959ea7773edc51dd4f91652ceadfb8dd0f3269f9c
Jul 01 19:32:34.567479 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.567432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.569499 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.569477    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls\") pod \"success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") " pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.671341 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.671271    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 01 19:32:34.798520 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.798488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:34.927397 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.927375    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"]
Jul 01 19:32:34.929111 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:32:34.929085    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00028cdb_adec_4d60_a662_dd3dfa6fba7e.slice/crio-c7960621d4e6176a938e5056d23b8b7172e48772ff6088f5e338c615a2dd7cdd WatchSource:0}: Error finding container c7960621d4e6176a938e5056d23b8b7172e48772ff6088f5e338c615a2dd7cdd: Status 404 returned error can't find the container with id c7960621d4e6176a938e5056d23b8b7172e48772ff6088f5e338c615a2dd7cdd
Jul 01 19:32:34.990247 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.990221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" event={"ID":"00028cdb-adec-4d60-a662-dd3dfa6fba7e","Type":"ContainerStarted","Data":"c7960621d4e6176a938e5056d23b8b7172e48772ff6088f5e338c615a2dd7cdd"}
Jul 01 19:32:34.992065 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.992042    2576 generic.go:356] "Generic (PLEG): container finished" podID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerID="25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735" exitCode=2
Jul 01 19:32:34.992171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.992114    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" event={"ID":"75bc0071-6882-493b-80ab-4ac0b0c2f01c","Type":"ContainerDied","Data":"25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735"}
Jul 01 19:32:34.993965 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.993945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" event={"ID":"c6ad769a-93dd-4cf1-b3c9-536391a9732c","Type":"ContainerStarted","Data":"d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b"}
Jul 01 19:32:34.994049 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.993998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" event={"ID":"c6ad769a-93dd-4cf1-b3c9-536391a9732c","Type":"ContainerStarted","Data":"6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b"}
Jul 01 19:32:34.994049 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.994032    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" event={"ID":"c6ad769a-93dd-4cf1-b3c9-536391a9732c","Type":"ContainerStarted","Data":"c415629e47fc1bd773d392c959ea7773edc51dd4f91652ceadfb8dd0f3269f9c"}
Jul 01 19:32:34.994122 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:34.994067    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:35.012364 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:35.012321    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podStartSLOduration=2.01230965 podStartE2EDuration="2.01230965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:35.010357277 +0000 UTC m=+681.769883990" watchObservedRunningTime="2026-07-01 19:32:35.01230965 +0000 UTC m=+681.771836371"
Jul 01 19:32:35.675976 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:35.675932    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 01 19:32:36.000644 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.000549    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" event={"ID":"00028cdb-adec-4d60-a662-dd3dfa6fba7e","Type":"ContainerStarted","Data":"5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8"}
Jul 01 19:32:36.000644 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.000628    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:36.001044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.000660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" event={"ID":"00028cdb-adec-4d60-a662-dd3dfa6fba7e","Type":"ContainerStarted","Data":"f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242"}
Jul 01 19:32:36.001044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.000840    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:36.001044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.000891    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:36.002045 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.002017    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 19:32:36.002146 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.002074    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 19:32:36.023006 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:36.022966    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podStartSLOduration=3.022947938 podStartE2EDuration="3.022947938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:32:36.02072861 +0000 UTC m=+682.780255331" watchObservedRunningTime="2026-07-01 19:32:36.022947938 +0000 UTC m=+682.782474666"
Jul 01 19:32:37.005066 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.005019    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 19:32:37.005531 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.005109    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 19:32:37.319726 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.319704    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:32:37.393970 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.393944    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-rbac-proxy-sar-config\") pod \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") "
Jul 01 19:32:37.394097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.394054    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnwpd\" (UniqueName: \"kubernetes.io/projected/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-api-access-lnwpd\") pod \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") "
Jul 01 19:32:37.394138 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.394101    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75bc0071-6882-493b-80ab-4ac0b0c2f01c-proxy-tls\") pod \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\" (UID: \"75bc0071-6882-493b-80ab-4ac0b0c2f01c\") "
Jul 01 19:32:37.394287 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.394256    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "75bc0071-6882-493b-80ab-4ac0b0c2f01c" (UID: "75bc0071-6882-493b-80ab-4ac0b0c2f01c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:32:37.394354 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.394342    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:32:37.396134 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.396107    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75bc0071-6882-493b-80ab-4ac0b0c2f01c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "75bc0071-6882-493b-80ab-4ac0b0c2f01c" (UID: "75bc0071-6882-493b-80ab-4ac0b0c2f01c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:32:37.396244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.396177    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-api-access-lnwpd" (OuterVolumeSpecName: "kube-api-access-lnwpd") pod "75bc0071-6882-493b-80ab-4ac0b0c2f01c" (UID: "75bc0071-6882-493b-80ab-4ac0b0c2f01c"). InnerVolumeSpecName "kube-api-access-lnwpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:32:37.495527 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.495444    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwpd\" (UniqueName: \"kubernetes.io/projected/75bc0071-6882-493b-80ab-4ac0b0c2f01c-kube-api-access-lnwpd\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:32:37.495527 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:37.495485    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/75bc0071-6882-493b-80ab-4ac0b0c2f01c-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:32:38.010627 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.010543    2576 generic.go:356] "Generic (PLEG): container finished" podID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerID="fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9" exitCode=0
Jul 01 19:32:38.010627 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.010599    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"
Jul 01 19:32:38.011055 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.010628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" event={"ID":"75bc0071-6882-493b-80ab-4ac0b0c2f01c","Type":"ContainerDied","Data":"fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9"}
Jul 01 19:32:38.011055 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.010686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s" event={"ID":"75bc0071-6882-493b-80ab-4ac0b0c2f01c","Type":"ContainerDied","Data":"6817df1978b1979ff7284ae6737779445f5ff5c694075d0674f2b35f85f99792"}
Jul 01 19:32:38.011055 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.010713    2576 scope.go:117] "RemoveContainer" containerID="25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735"
Jul 01 19:32:38.019185 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.019168    2576 scope.go:117] "RemoveContainer" containerID="fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9"
Jul 01 19:32:38.026748 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.026727    2576 scope.go:117] "RemoveContainer" containerID="25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735"
Jul 01 19:32:38.027030 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:32:38.027007    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735\": container with ID starting with 25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735 not found: ID does not exist" containerID="25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735"
Jul 01 19:32:38.027129 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.027042    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735"} err="failed to get container status \"25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735\": rpc error: code = NotFound desc = could not find container \"25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735\": container with ID starting with 25f14d116cf32f339188c19796715f4bf9afca50a2568aee779221b9896d4735 not found: ID does not exist"
Jul 01 19:32:38.027129 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.027074    2576 scope.go:117] "RemoveContainer" containerID="fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9"
Jul 01 19:32:38.027368 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:32:38.027348    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9\": container with ID starting with fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9 not found: ID does not exist" containerID="fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9"
Jul 01 19:32:38.027489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.027467    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9"} err="failed to get container status \"fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9\": rpc error: code = NotFound desc = could not find container \"fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9\": container with ID starting with fca46f8966545b9c9d546b4a170273dd15d1e7f9b068e2a8abb802549c4f8ae9 not found: ID does not exist"
Jul 01 19:32:38.029560 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.029542    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"]
Jul 01 19:32:38.033957 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:38.033936    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-79b53-predictor-64ff7f459d-xjf2s"]
Jul 01 19:32:39.751530 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:39.751483    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" path="/var/lib/kubelet/pods/75bc0071-6882-493b-80ab-4ac0b0c2f01c/volumes"
Jul 01 19:32:40.022571 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.022497    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerID="dd8f0a9d1c7c9c03756cac1af1c8b4c9ab6b2fd7a822b703883749edfd6e8c28" exitCode=0
Jul 01 19:32:40.022704 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.022571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" event={"ID":"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a","Type":"ContainerDied","Data":"dd8f0a9d1c7c9c03756cac1af1c8b4c9ab6b2fd7a822b703883749edfd6e8c28"}
Jul 01 19:32:40.055745 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.055720    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:32:40.119882 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.119855    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-rbac-proxy-sar-config\") pod \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") "
Jul 01 19:32:40.120009 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.119907    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-proxy-tls\") pod \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") "
Jul 01 19:32:40.120009 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.119959    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64rkw\" (UniqueName: \"kubernetes.io/projected/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-api-access-64rkw\") pod \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\" (UID: \"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a\") "
Jul 01 19:32:40.120271 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.120248    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" (UID: "f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:32:40.121967 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.121946    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" (UID: "f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:32:40.122099 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.122080    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-api-access-64rkw" (OuterVolumeSpecName: "kube-api-access-64rkw") pod "f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" (UID: "f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a"). InnerVolumeSpecName "kube-api-access-64rkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:32:40.221561 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.221534    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:32:40.221561 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.221559    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:32:40.221700 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:40.221568    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rkw\" (UniqueName: \"kubernetes.io/projected/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a-kube-api-access-64rkw\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:32:41.029079 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:41.029050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk" event={"ID":"f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a","Type":"ContainerDied","Data":"ca6d6cd329fb3a496e805963937fc35b67796842fdff38a4ffec5f1dcb29acaa"}
Jul 01 19:32:41.029079 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:41.029070    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"
Jul 01 19:32:41.029687 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:41.029124    2576 scope.go:117] "RemoveContainer" containerID="6d3c678d342109c5e983560e61a6f21df364dfd0b980f7035c3bbba2c5b75f4d"
Jul 01 19:32:41.037771 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:41.037752    2576 scope.go:117] "RemoveContainer" containerID="dd8f0a9d1c7c9c03756cac1af1c8b4c9ab6b2fd7a822b703883749edfd6e8c28"
Jul 01 19:32:41.055028 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:41.055009    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"]
Jul 01 19:32:41.058625 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:41.058606    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-79b53-predictor-58774dd6b7-tzcmk"]
Jul 01 19:32:41.752941 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:41.752905    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" path="/var/lib/kubelet/pods/f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a/volumes"
Jul 01 19:32:42.010101 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:42.010023    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:32:42.010395 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:42.010376    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:32:42.010509 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:42.010473    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 19:32:42.010902 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:42.010881    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 19:32:52.010944 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:52.010901    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 19:32:52.011326 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:32:52.010901    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 19:33:02.011278 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:02.011232    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 19:33:02.011684 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:02.011234    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 19:33:12.011443 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:12.011355    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 19:33:12.011894 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:12.011356    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 01 19:33:13.919774 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.919740    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"]
Jul 01 19:33:13.920200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920151    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container"
Jul 01 19:33:13.920200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920164    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container"
Jul 01 19:33:13.920200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920182    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kube-rbac-proxy"
Jul 01 19:33:13.920200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920188    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kube-rbac-proxy"
Jul 01 19:33:13.920200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920197    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container"
Jul 01 19:33:13.920200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920203    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container"
Jul 01 19:33:13.920489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920235    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kube-rbac-proxy"
Jul 01 19:33:13.920489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920245    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kube-rbac-proxy"
Jul 01 19:33:13.920489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920323    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kube-rbac-proxy"
Jul 01 19:33:13.920489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920336    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f04157-2a0a-4c48-93e8-7c17c1ee0b8a" containerName="kserve-container"
Jul 01 19:33:13.920489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920351    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kube-rbac-proxy"
Jul 01 19:33:13.920489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.920361    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="75bc0071-6882-493b-80ab-4ac0b0c2f01c" containerName="kserve-container"
Jul 01 19:33:13.925133 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.925114    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:13.929226 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.929205    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d109c-predictor-serving-cert\""
Jul 01 19:33:13.929377 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.929230    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d109c-kube-rbac-proxy-sar-config\""
Jul 01 19:33:13.939834 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.939806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"]
Jul 01 19:33:13.987793 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.987762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46fc9c-d916-4629-9fce-f865f714f516-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:13.987928 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.987819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46fc9c-d916-4629-9fce-f865f714f516-proxy-tls\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:13.987928 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:13.987871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl6zs\" (UniqueName: \"kubernetes.io/projected/aa46fc9c-d916-4629-9fce-f865f714f516-kube-api-access-hl6zs\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.088546 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.088507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46fc9c-d916-4629-9fce-f865f714f516-proxy-tls\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.088725 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.088555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl6zs\" (UniqueName: \"kubernetes.io/projected/aa46fc9c-d916-4629-9fce-f865f714f516-kube-api-access-hl6zs\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.088725 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.088639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46fc9c-d916-4629-9fce-f865f714f516-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.089240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.089214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46fc9c-d916-4629-9fce-f865f714f516-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.091045 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.091004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46fc9c-d916-4629-9fce-f865f714f516-proxy-tls\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.096628 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.096605    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"]
Jul 01 19:33:14.100300 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.100281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.107746 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.107728    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d109c-predictor-serving-cert\""
Jul 01 19:33:14.109365 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.109344    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d109c-kube-rbac-proxy-sar-config\""
Jul 01 19:33:14.110089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.110069    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl6zs\" (UniqueName: \"kubernetes.io/projected/aa46fc9c-d916-4629-9fce-f865f714f516-kube-api-access-hl6zs\") pod \"success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") " pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.134310 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.134284    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"]
Jul 01 19:33:14.189409 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.189380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a910e61-15e1-46c0-82da-d2781902c582-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.189568 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.189469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a910e61-15e1-46c0-82da-d2781902c582-proxy-tls\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.189568 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.189516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btbr\" (UniqueName: \"kubernetes.io/projected/8a910e61-15e1-46c0-82da-d2781902c582-kube-api-access-9btbr\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.239402 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.239372    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:14.290443 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.290414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a910e61-15e1-46c0-82da-d2781902c582-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.290597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.290506    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a910e61-15e1-46c0-82da-d2781902c582-proxy-tls\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.290597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.290540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btbr\" (UniqueName: \"kubernetes.io/projected/8a910e61-15e1-46c0-82da-d2781902c582-kube-api-access-9btbr\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.291159 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.291129    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a910e61-15e1-46c0-82da-d2781902c582-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.294274 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.293672    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a910e61-15e1-46c0-82da-d2781902c582-proxy-tls\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.301192 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.300869    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btbr\" (UniqueName: \"kubernetes.io/projected/8a910e61-15e1-46c0-82da-d2781902c582-kube-api-access-9btbr\") pod \"error-404-isvc-d109c-predictor-5d9b9bf885-t4frr\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") " pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.383573 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.383539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"]
Jul 01 19:33:14.384662 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:33:14.384631    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa46fc9c_d916_4629_9fce_f865f714f516.slice/crio-71f3cc9bc353a90611d1af8ea99c38beea942d165f63c1c6172a0886e591f7c6 WatchSource:0}: Error finding container 71f3cc9bc353a90611d1af8ea99c38beea942d165f63c1c6172a0886e591f7c6: Status 404 returned error can't find the container with id 71f3cc9bc353a90611d1af8ea99c38beea942d165f63c1c6172a0886e591f7c6
Jul 01 19:33:14.410839 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.410814    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:14.550517 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:14.550489    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"]
Jul 01 19:33:14.551505 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:33:14.551478    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a910e61_15e1_46c0_82da_d2781902c582.slice/crio-2646613e1f1aac45be10ba24e78ff3bb95574ca4f13a04e72f33dc7ec8a57e17 WatchSource:0}: Error finding container 2646613e1f1aac45be10ba24e78ff3bb95574ca4f13a04e72f33dc7ec8a57e17: Status 404 returned error can't find the container with id 2646613e1f1aac45be10ba24e78ff3bb95574ca4f13a04e72f33dc7ec8a57e17
Jul 01 19:33:15.201157 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.201126    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" event={"ID":"aa46fc9c-d916-4629-9fce-f865f714f516","Type":"ContainerStarted","Data":"0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3"}
Jul 01 19:33:15.201597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.201323    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:15.201597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.201363    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" event={"ID":"aa46fc9c-d916-4629-9fce-f865f714f516","Type":"ContainerStarted","Data":"fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa"}
Jul 01 19:33:15.201597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.201387    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:15.201597 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.201412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" event={"ID":"aa46fc9c-d916-4629-9fce-f865f714f516","Type":"ContainerStarted","Data":"71f3cc9bc353a90611d1af8ea99c38beea942d165f63c1c6172a0886e591f7c6"}
Jul 01 19:33:15.202794 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.202764    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 19:33:15.203106 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.203086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" event={"ID":"8a910e61-15e1-46c0-82da-d2781902c582","Type":"ContainerStarted","Data":"2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d"}
Jul 01 19:33:15.203196 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.203126    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" event={"ID":"8a910e61-15e1-46c0-82da-d2781902c582","Type":"ContainerStarted","Data":"966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9"}
Jul 01 19:33:15.203196 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.203150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" event={"ID":"8a910e61-15e1-46c0-82da-d2781902c582","Type":"ContainerStarted","Data":"2646613e1f1aac45be10ba24e78ff3bb95574ca4f13a04e72f33dc7ec8a57e17"}
Jul 01 19:33:15.203312 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.203227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:15.220195 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.220151    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podStartSLOduration=2.220139272 podStartE2EDuration="2.220139272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:33:15.218740857 +0000 UTC m=+721.978267596" watchObservedRunningTime="2026-07-01 19:33:15.220139272 +0000 UTC m=+721.979665993"
Jul 01 19:33:15.239255 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:15.239220    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podStartSLOduration=1.239207833 podStartE2EDuration="1.239207833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:33:15.237711612 +0000 UTC m=+721.997238334" watchObservedRunningTime="2026-07-01 19:33:15.239207833 +0000 UTC m=+721.998734554"
Jul 01 19:33:16.208509 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:16.208484    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:16.208865 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:16.208473    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 19:33:16.210006 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:16.209978    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 19:33:17.212912 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:17.212875    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 19:33:21.213659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:21.213631    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:33:21.214057 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:21.214024    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 19:33:22.011640 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:22.011614    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:33:22.011805 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:22.011709    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:33:22.217502 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:22.217474    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:33:22.217853 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:22.217829    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 19:33:31.214849 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:31.214808    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 19:33:32.218458 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:32.218420    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 19:33:41.215005 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:41.214962    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 19:33:42.218528 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:42.218492    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 19:33:51.214779 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:51.214736    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 19:33:52.218096 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:33:52.218059    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 01 19:34:01.215429 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:34:01.215394    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:34:02.218626 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:34:02.218590    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:36:20.366749 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:36:20.366648    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:36:20.375205 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:36:20.375176    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:41:20.407244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:20.407120    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:41:20.411225 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:20.410764    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:41:48.851330 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.851290    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"]
Jul 01 19:41:48.851793 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.851596    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container" containerID="cri-o://5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8" gracePeriod=30
Jul 01 19:41:48.852380 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.852073    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kube-rbac-proxy" containerID="cri-o://f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242" gracePeriod=30
Jul 01 19:41:48.861426 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.861400    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"]
Jul 01 19:41:48.865196 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.865178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:48.867652 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.867634    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-44b6c-predictor-serving-cert\""
Jul 01 19:41:48.867750 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.867672    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-44b6c-kube-rbac-proxy-sar-config\""
Jul 01 19:41:48.875265 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.875240    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"]
Jul 01 19:41:48.951717 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.951687    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"]
Jul 01 19:41:48.955441 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.955419    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:48.958464 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.958433    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-44b6c-predictor-serving-cert\""
Jul 01 19:41:48.958800 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.958784    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-44b6c-kube-rbac-proxy-sar-config\""
Jul 01 19:41:48.966852 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.966829    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"]
Jul 01 19:41:48.968964 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.968940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:48.969055 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.968995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sh2w\" (UniqueName: \"kubernetes.io/projected/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-api-access-9sh2w\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:48.969117 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:48.969102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.016614 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.016582    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"]
Jul 01 19:41:49.016813 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.016780    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" containerID="cri-o://6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b" gracePeriod=30
Jul 01 19:41:49.016928 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.016807    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kube-rbac-proxy" containerID="cri-o://d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b" gracePeriod=30
Jul 01 19:41:49.070850 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.070805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ca7119-e603-4b2d-93f8-2eea92ef0800-proxy-tls\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.070973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.070874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.070973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.070951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgsc\" (UniqueName: \"kubernetes.io/projected/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-api-access-4vgsc\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.071081 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.070998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.071081 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.071039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.071185 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.071082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sh2w\" (UniqueName: \"kubernetes.io/projected/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-api-access-9sh2w\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.072181 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.072150    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.072325 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:41:49.072308    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-serving-cert: secret "success-200-isvc-44b6c-predictor-serving-cert" not found
Jul 01 19:41:49.072397 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:41:49.072385    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls podName:c217f2fc-cd4e-4286-a5fd-961a1809ca4f nodeName:}" failed. No retries permitted until 2026-07-01 19:41:49.572360222 +0000 UTC m=+1236.331886921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls") pod "success-200-isvc-44b6c-predictor-788df44c68-prgzl" (UID: "c217f2fc-cd4e-4286-a5fd-961a1809ca4f") : secret "success-200-isvc-44b6c-predictor-serving-cert" not found
Jul 01 19:41:49.089101 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.089073    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sh2w\" (UniqueName: \"kubernetes.io/projected/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-api-access-9sh2w\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.171713 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.171681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ca7119-e603-4b2d-93f8-2eea92ef0800-proxy-tls\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.171840 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.171734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgsc\" (UniqueName: \"kubernetes.io/projected/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-api-access-4vgsc\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.171840 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.171798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.172537 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.172516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.174138 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.174120    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ca7119-e603-4b2d-93f8-2eea92ef0800-proxy-tls\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.187191 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.187155    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgsc\" (UniqueName: \"kubernetes.io/projected/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-api-access-4vgsc\") pod \"error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") " pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.267118 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.267086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.403431 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.403399    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"]
Jul 01 19:41:49.405209 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:41:49.405188    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ca7119_e603_4b2d_93f8_2eea92ef0800.slice/crio-94063cde7a9fc4a88c9efae40a8a5037502a32ce7f42dcea4071401617ae9818 WatchSource:0}: Error finding container 94063cde7a9fc4a88c9efae40a8a5037502a32ce7f42dcea4071401617ae9818: Status 404 returned error can't find the container with id 94063cde7a9fc4a88c9efae40a8a5037502a32ce7f42dcea4071401617ae9818
Jul 01 19:41:49.575189 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.575161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.577248 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.577231    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls\") pod \"success-200-isvc-44b6c-predictor-788df44c68-prgzl\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") " pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.678922 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.678845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" event={"ID":"c3ca7119-e603-4b2d-93f8-2eea92ef0800","Type":"ContainerStarted","Data":"74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8"}
Jul 01 19:41:49.678922 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.678908    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" event={"ID":"c3ca7119-e603-4b2d-93f8-2eea92ef0800","Type":"ContainerStarted","Data":"99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248"}
Jul 01 19:41:49.679108 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.678945    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.679108 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.678976    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:49.679108 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.679002    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" event={"ID":"c3ca7119-e603-4b2d-93f8-2eea92ef0800","Type":"ContainerStarted","Data":"94063cde7a9fc4a88c9efae40a8a5037502a32ce7f42dcea4071401617ae9818"}
Jul 01 19:41:49.680731 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.680705    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 19:41:49.680831 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.680738    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerID="d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b" exitCode=2
Jul 01 19:41:49.680831 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.680799    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" event={"ID":"c6ad769a-93dd-4cf1-b3c9-536391a9732c","Type":"ContainerDied","Data":"d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b"}
Jul 01 19:41:49.682566 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.682543    2576 generic.go:356] "Generic (PLEG): container finished" podID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerID="f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242" exitCode=2
Jul 01 19:41:49.682638 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.682596    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" event={"ID":"00028cdb-adec-4d60-a662-dd3dfa6fba7e","Type":"ContainerDied","Data":"f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242"}
Jul 01 19:41:49.697144 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.697102    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podStartSLOduration=1.697090754 podStartE2EDuration="1.697090754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:41:49.695997754 +0000 UTC m=+1236.455524497" watchObservedRunningTime="2026-07-01 19:41:49.697090754 +0000 UTC m=+1236.456617474"
Jul 01 19:41:49.775901 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.775874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:49.907604 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:49.907576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"]
Jul 01 19:41:49.909850 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:41:49.909823    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc217f2fc_cd4e_4286_a5fd_961a1809ca4f.slice/crio-e19487c9e8c0a9b9194a7b77137a3193e4824e59635937d74afb72b0b94826bb WatchSource:0}: Error finding container e19487c9e8c0a9b9194a7b77137a3193e4824e59635937d74afb72b0b94826bb: Status 404 returned error can't find the container with id e19487c9e8c0a9b9194a7b77137a3193e4824e59635937d74afb72b0b94826bb
Jul 01 19:41:50.688834 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:50.688795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" event={"ID":"c217f2fc-cd4e-4286-a5fd-961a1809ca4f","Type":"ContainerStarted","Data":"1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca"}
Jul 01 19:41:50.689056 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:50.688854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" event={"ID":"c217f2fc-cd4e-4286-a5fd-961a1809ca4f","Type":"ContainerStarted","Data":"2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a"}
Jul 01 19:41:50.689056 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:50.688877    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" event={"ID":"c217f2fc-cd4e-4286-a5fd-961a1809ca4f","Type":"ContainerStarted","Data":"e19487c9e8c0a9b9194a7b77137a3193e4824e59635937d74afb72b0b94826bb"}
Jul 01 19:41:50.689172 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:50.689055    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:50.689286 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:50.689257    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 19:41:50.707658 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:50.707608    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podStartSLOduration=2.7075952660000002 podStartE2EDuration="2.707595266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:41:50.706724936 +0000 UTC m=+1237.466251669" watchObservedRunningTime="2026-07-01 19:41:50.707595266 +0000 UTC m=+1237.467121979"
Jul 01 19:41:51.693962 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:51.693919    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:51.695515 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:51.695484    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 19:41:51.910355 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:51.910329    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:41:51.998489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:51.998403    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls\") pod \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") "
Jul 01 19:41:51.998489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:51.998474    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpw89\" (UniqueName: \"kubernetes.io/projected/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-api-access-fpw89\") pod \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") "
Jul 01 19:41:51.998653 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:51.998536    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-rbac-proxy-sar-config\") pod \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\" (UID: \"00028cdb-adec-4d60-a662-dd3dfa6fba7e\") "
Jul 01 19:41:51.998913 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:51.998887    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00028cdb-adec-4d60-a662-dd3dfa6fba7e" (UID: "00028cdb-adec-4d60-a662-dd3dfa6fba7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:41:52.000394 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.000371    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00028cdb-adec-4d60-a662-dd3dfa6fba7e" (UID: "00028cdb-adec-4d60-a662-dd3dfa6fba7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:41:52.000529 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.000508    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-api-access-fpw89" (OuterVolumeSpecName: "kube-api-access-fpw89") pod "00028cdb-adec-4d60-a662-dd3dfa6fba7e" (UID: "00028cdb-adec-4d60-a662-dd3dfa6fba7e"). InnerVolumeSpecName "kube-api-access-fpw89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:41:52.005840 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.005805    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 01 19:41:52.011382 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.011354    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 01 19:41:52.099630 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.099606    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpw89\" (UniqueName: \"kubernetes.io/projected/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-api-access-fpw89\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:41:52.099630 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.099627    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00028cdb-adec-4d60-a662-dd3dfa6fba7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:41:52.099759 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.099637    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00028cdb-adec-4d60-a662-dd3dfa6fba7e-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:41:52.699984 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.699952    2576 generic.go:356] "Generic (PLEG): container finished" podID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerID="5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8" exitCode=0
Jul 01 19:41:52.700373 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.699993    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" event={"ID":"00028cdb-adec-4d60-a662-dd3dfa6fba7e","Type":"ContainerDied","Data":"5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8"}
Jul 01 19:41:52.700373 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.700058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl" event={"ID":"00028cdb-adec-4d60-a662-dd3dfa6fba7e","Type":"ContainerDied","Data":"c7960621d4e6176a938e5056d23b8b7172e48772ff6088f5e338c615a2dd7cdd"}
Jul 01 19:41:52.700373 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.700086    2576 scope.go:117] "RemoveContainer" containerID="f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242"
Jul 01 19:41:52.700373 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.700008    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"
Jul 01 19:41:52.700598 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.700525    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 19:41:52.710337 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.710316    2576 scope.go:117] "RemoveContainer" containerID="5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8"
Jul 01 19:41:52.719376 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.719360    2576 scope.go:117] "RemoveContainer" containerID="f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242"
Jul 01 19:41:52.719623 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:41:52.719604    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242\": container with ID starting with f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242 not found: ID does not exist" containerID="f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242"
Jul 01 19:41:52.719703 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.719632    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242"} err="failed to get container status \"f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242\": rpc error: code = NotFound desc = could not find container \"f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242\": container with ID starting with f9a13ba00ee7702b223437bb5ccd8fe06e35ae03fdc67642f99597dc7152f242 not found: ID does not exist"
Jul 01 19:41:52.719703 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.719648    2576 scope.go:117] "RemoveContainer" containerID="5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8"
Jul 01 19:41:52.719864 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:41:52.719849    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8\": container with ID starting with 5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8 not found: ID does not exist" containerID="5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8"
Jul 01 19:41:52.719908 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.719868    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8"} err="failed to get container status \"5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8\": rpc error: code = NotFound desc = could not find container \"5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8\": container with ID starting with 5209b7bb7d90ad4f6cefd46134e41fe7c8c837197d5f976a36b452d4c6d038a8 not found: ID does not exist"
Jul 01 19:41:52.724770 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.724745    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"]
Jul 01 19:41:52.726734 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:52.726712    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c6c1-predictor-6c95f7b7dd-wx4sl"]
Jul 01 19:41:53.751355 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:53.751321    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" path="/var/lib/kubelet/pods/00028cdb-adec-4d60-a662-dd3dfa6fba7e/volumes"
Jul 01 19:41:55.675863 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.675838    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:41:55.694407 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.694337    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:41:55.694889 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.694860    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 19:41:55.718269 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.718237    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerID="6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b" exitCode=0
Jul 01 19:41:55.718374 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.718298    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"
Jul 01 19:41:55.718425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.718299    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" event={"ID":"c6ad769a-93dd-4cf1-b3c9-536391a9732c","Type":"ContainerDied","Data":"6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b"}
Jul 01 19:41:55.718485 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.718419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8" event={"ID":"c6ad769a-93dd-4cf1-b3c9-536391a9732c","Type":"ContainerDied","Data":"c415629e47fc1bd773d392c959ea7773edc51dd4f91652ceadfb8dd0f3269f9c"}
Jul 01 19:41:55.718607 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.718480    2576 scope.go:117] "RemoveContainer" containerID="d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b"
Jul 01 19:41:55.726745 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.726715    2576 scope.go:117] "RemoveContainer" containerID="6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b"
Jul 01 19:41:55.730945 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.730921    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ad769a-93dd-4cf1-b3c9-536391a9732c-proxy-tls\") pod \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") "
Jul 01 19:41:55.731045 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.730991    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkmzw\" (UniqueName: \"kubernetes.io/projected/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-api-access-jkmzw\") pod \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") "
Jul 01 19:41:55.731045 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.731034    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-rbac-proxy-sar-config\") pod \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\" (UID: \"c6ad769a-93dd-4cf1-b3c9-536391a9732c\") "
Jul 01 19:41:55.731404 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.731377    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6ad769a-93dd-4cf1-b3c9-536391a9732c" (UID: "c6ad769a-93dd-4cf1-b3c9-536391a9732c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:41:55.733330 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.733304    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-api-access-jkmzw" (OuterVolumeSpecName: "kube-api-access-jkmzw") pod "c6ad769a-93dd-4cf1-b3c9-536391a9732c" (UID: "c6ad769a-93dd-4cf1-b3c9-536391a9732c"). InnerVolumeSpecName "kube-api-access-jkmzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:41:55.733330 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.733307    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ad769a-93dd-4cf1-b3c9-536391a9732c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6ad769a-93dd-4cf1-b3c9-536391a9732c" (UID: "c6ad769a-93dd-4cf1-b3c9-536391a9732c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:41:55.734411 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.734396    2576 scope.go:117] "RemoveContainer" containerID="d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b"
Jul 01 19:41:55.734761 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:41:55.734742    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b\": container with ID starting with d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b not found: ID does not exist" containerID="d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b"
Jul 01 19:41:55.734829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.734766    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b"} err="failed to get container status \"d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b\": rpc error: code = NotFound desc = could not find container \"d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b\": container with ID starting with d93c819c5ed75ad2c560b40750cebd5405395aea203a595b8dbb8cac69b71c2b not found: ID does not exist"
Jul 01 19:41:55.734829 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.734781    2576 scope.go:117] "RemoveContainer" containerID="6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b"
Jul 01 19:41:55.735015 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:41:55.734997    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b\": container with ID starting with 6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b not found: ID does not exist" containerID="6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b"
Jul 01 19:41:55.735057 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.735022    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b"} err="failed to get container status \"6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b\": rpc error: code = NotFound desc = could not find container \"6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b\": container with ID starting with 6f11821ab578632d2c00d6788f1c4b25fe2122a832de7ede00567ebd7f956e0b not found: ID does not exist"
Jul 01 19:41:55.832308 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.832283    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ad769a-93dd-4cf1-b3c9-536391a9732c-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:41:55.832308 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.832308    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkmzw\" (UniqueName: \"kubernetes.io/projected/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-api-access-jkmzw\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:41:55.832488 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:55.832322    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ad769a-93dd-4cf1-b3c9-536391a9732c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:41:56.035090 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:56.035063    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"]
Jul 01 19:41:56.041574 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:56.041545    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-6c6c1-predictor-5cd754887c-fnxm8"]
Jul 01 19:41:57.704670 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:57.704639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:41:57.705087 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:57.705063    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 19:41:57.750744 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:41:57.750717    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" path="/var/lib/kubelet/pods/c6ad769a-93dd-4cf1-b3c9-536391a9732c/volumes"
Jul 01 19:42:05.695129 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:05.695091    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 19:42:07.705677 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:07.705599    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 19:42:15.695351 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:15.695310    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 19:42:17.705987 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:17.705940    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 19:42:25.695677 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:25.695637    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 01 19:42:27.705696 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:27.705655    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 01 19:42:28.743692 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.743652    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"]
Jul 01 19:42:28.744152 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.743952    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container" containerID="cri-o://966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9" gracePeriod=30
Jul 01 19:42:28.744152 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.744035    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kube-rbac-proxy" containerID="cri-o://2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d" gracePeriod=30
Jul 01 19:42:28.807547 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.807512    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"]
Jul 01 19:42:28.807856 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.807827    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" containerID="cri-o://fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa" gracePeriod=30
Jul 01 19:42:28.807964 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.807878    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kube-rbac-proxy" containerID="cri-o://0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3" gracePeriod=30
Jul 01 19:42:28.822345 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822320    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"]
Jul 01 19:42:28.822799 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822780    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container"
Jul 01 19:42:28.822900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822802    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container"
Jul 01 19:42:28.822900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822828    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container"
Jul 01 19:42:28.822900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822837    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container"
Jul 01 19:42:28.822900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822858    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kube-rbac-proxy"
Jul 01 19:42:28.822900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822868    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kube-rbac-proxy"
Jul 01 19:42:28.822900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822880    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kube-rbac-proxy"
Jul 01 19:42:28.822900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822888    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kube-rbac-proxy"
Jul 01 19:42:28.823245 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.822994    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kserve-container"
Jul 01 19:42:28.823245 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.823010    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kube-rbac-proxy"
Jul 01 19:42:28.823245 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.823022    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="00028cdb-adec-4d60-a662-dd3dfa6fba7e" containerName="kserve-container"
Jul 01 19:42:28.823245 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.823033    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ad769a-93dd-4cf1-b3c9-536391a9732c" containerName="kube-rbac-proxy"
Jul 01 19:42:28.827662 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.827643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:28.830244 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.830225    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7f91-kube-rbac-proxy-sar-config\""
Jul 01 19:42:28.830866 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.830841    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7f91-predictor-serving-cert\""
Jul 01 19:42:28.835803 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.835783    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"]
Jul 01 19:42:28.882200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.882173    2576 generic.go:356] "Generic (PLEG): container finished" podID="8a910e61-15e1-46c0-82da-d2781902c582" containerID="2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d" exitCode=2
Jul 01 19:42:28.882359 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.882249    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" event={"ID":"8a910e61-15e1-46c0-82da-d2781902c582","Type":"ContainerDied","Data":"2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d"}
Jul 01 19:42:28.911443 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.911412    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"]
Jul 01 19:42:28.914407 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.914384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-proxy-tls\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:28.914553 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.914472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdw9\" (UniqueName: \"kubernetes.io/projected/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-api-access-jqdw9\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:28.914605 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.914578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:28.915171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.915152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:28.918360 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.918338    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7f91-kube-rbac-proxy-sar-config\""
Jul 01 19:42:28.920672 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.920650    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7f91-predictor-serving-cert\""
Jul 01 19:42:28.926236 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:28.926208    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"]
Jul 01 19:42:29.015102 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.015014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdw9\" (UniqueName: \"kubernetes.io/projected/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-api-access-jqdw9\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.015102 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.015062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.015102 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.015082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97x5j\" (UniqueName: \"kubernetes.io/projected/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-api-access-97x5j\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.015382 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.015117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.015382 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.015237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.015382 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.015330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-proxy-tls\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.015894 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.015866    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.017848 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.017828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-proxy-tls\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.024050 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.024031    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdw9\" (UniqueName: \"kubernetes.io/projected/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-api-access-jqdw9\") pod \"success-200-isvc-f7f91-predictor-5974f76cdd-qs7px\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") " pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.115786 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.115758    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.115786 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.115791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97x5j\" (UniqueName: \"kubernetes.io/projected/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-api-access-97x5j\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.116007 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.115813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.116007 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:29.115920    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-serving-cert: secret "error-404-isvc-f7f91-predictor-serving-cert" not found
Jul 01 19:42:29.116007 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:29.115973    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls podName:492f1bb4-3f05-44d8-bb47-1a1b044cf510 nodeName:}" failed. No retries permitted until 2026-07-01 19:42:29.615955964 +0000 UTC m=+1276.375482680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls") pod "error-404-isvc-f7f91-predictor-69c44d6649-89l57" (UID: "492f1bb4-3f05-44d8-bb47-1a1b044cf510") : secret "error-404-isvc-f7f91-predictor-serving-cert" not found
Jul 01 19:42:29.116442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.116421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.131089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.131060    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97x5j\" (UniqueName: \"kubernetes.io/projected/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-api-access-97x5j\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.139434 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.139409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.278664 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.278639    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"]
Jul 01 19:42:29.280422 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:42:29.280397    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334395e0_b09e_41b9_bb5c_4eb4b371eaa8.slice/crio-1dc28690af091ecfa9639d3a482d1bcb4cbeff191af7f9e9a9bef5494f967758 WatchSource:0}: Error finding container 1dc28690af091ecfa9639d3a482d1bcb4cbeff191af7f9e9a9bef5494f967758: Status 404 returned error can't find the container with id 1dc28690af091ecfa9639d3a482d1bcb4cbeff191af7f9e9a9bef5494f967758
Jul 01 19:42:29.618725 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.618694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.621014 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.620993    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls\") pod \"error-404-isvc-f7f91-predictor-69c44d6649-89l57\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") " pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.827442 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.827411    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:29.892844 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.892808    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" event={"ID":"334395e0-b09e-41b9-bb5c-4eb4b371eaa8","Type":"ContainerStarted","Data":"1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33"}
Jul 01 19:42:29.893004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.892876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" event={"ID":"334395e0-b09e-41b9-bb5c-4eb4b371eaa8","Type":"ContainerStarted","Data":"3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11"}
Jul 01 19:42:29.893004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.892916    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:29.893004 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.892948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" event={"ID":"334395e0-b09e-41b9-bb5c-4eb4b371eaa8","Type":"ContainerStarted","Data":"1dc28690af091ecfa9639d3a482d1bcb4cbeff191af7f9e9a9bef5494f967758"}
Jul 01 19:42:29.894744 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.894715    2576 generic.go:356] "Generic (PLEG): container finished" podID="aa46fc9c-d916-4629-9fce-f865f714f516" containerID="0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3" exitCode=2
Jul 01 19:42:29.894864 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.894781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" event={"ID":"aa46fc9c-d916-4629-9fce-f865f714f516","Type":"ContainerDied","Data":"0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3"}
Jul 01 19:42:29.917823 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.917781    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podStartSLOduration=1.917762893 podStartE2EDuration="1.917762893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:42:29.916720155 +0000 UTC m=+1276.676246878" watchObservedRunningTime="2026-07-01 19:42:29.917762893 +0000 UTC m=+1276.677289614"
Jul 01 19:42:29.969604 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:29.969576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"]
Jul 01 19:42:29.971472 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:42:29.971429    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492f1bb4_3f05_44d8_bb47_1a1b044cf510.slice/crio-def9737aa38991faa1b62c571a6d83d64caf0e8b671e33c444f89730bc11ff1d WatchSource:0}: Error finding container def9737aa38991faa1b62c571a6d83d64caf0e8b671e33c444f89730bc11ff1d: Status 404 returned error can't find the container with id def9737aa38991faa1b62c571a6d83d64caf0e8b671e33c444f89730bc11ff1d
Jul 01 19:42:30.901929 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.901889    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" event={"ID":"492f1bb4-3f05-44d8-bb47-1a1b044cf510","Type":"ContainerStarted","Data":"6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4"}
Jul 01 19:42:30.902398 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.902283    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:30.902398 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.902338    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" event={"ID":"492f1bb4-3f05-44d8-bb47-1a1b044cf510","Type":"ContainerStarted","Data":"dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626"}
Jul 01 19:42:30.902398 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.902366    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:30.902398 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.902383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" event={"ID":"492f1bb4-3f05-44d8-bb47-1a1b044cf510","Type":"ContainerStarted","Data":"def9737aa38991faa1b62c571a6d83d64caf0e8b671e33c444f89730bc11ff1d"}
Jul 01 19:42:30.902657 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.902403    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:30.903370 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.903341    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 01 19:42:30.903489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.903440    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 01 19:42:30.922487 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:30.922430    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podStartSLOduration=2.922417488 podStartE2EDuration="2.922417488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:42:30.921754688 +0000 UTC m=+1277.681281426" watchObservedRunningTime="2026-07-01 19:42:30.922417488 +0000 UTC m=+1277.681944206"
Jul 01 19:42:31.209718 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.209617    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 01 19:42:31.214213 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.214187    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 01 19:42:31.903440 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.903410    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:42:31.909555 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.909527    2576 generic.go:356] "Generic (PLEG): container finished" podID="8a910e61-15e1-46c0-82da-d2781902c582" containerID="966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9" exitCode=0
Jul 01 19:42:31.909673 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.909575    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"
Jul 01 19:42:31.909673 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.909604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" event={"ID":"8a910e61-15e1-46c0-82da-d2781902c582","Type":"ContainerDied","Data":"966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9"}
Jul 01 19:42:31.909673 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.909661    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr" event={"ID":"8a910e61-15e1-46c0-82da-d2781902c582","Type":"ContainerDied","Data":"2646613e1f1aac45be10ba24e78ff3bb95574ca4f13a04e72f33dc7ec8a57e17"}
Jul 01 19:42:31.909805 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.909687    2576 scope.go:117] "RemoveContainer" containerID="2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d"
Jul 01 19:42:31.910060 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.910032    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 01 19:42:31.910170 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.910113    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 01 19:42:31.917745 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.917729    2576 scope.go:117] "RemoveContainer" containerID="966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9"
Jul 01 19:42:31.926635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.926619    2576 scope.go:117] "RemoveContainer" containerID="2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d"
Jul 01 19:42:31.926901 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:31.926873    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d\": container with ID starting with 2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d not found: ID does not exist" containerID="2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d"
Jul 01 19:42:31.926991 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.926907    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d"} err="failed to get container status \"2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d\": rpc error: code = NotFound desc = could not find container \"2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d\": container with ID starting with 2cdb010b4f883fc99c0b129d2239125312921ef205e1d121730693de7f6fee4d not found: ID does not exist"
Jul 01 19:42:31.926991 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.926922    2576 scope.go:117] "RemoveContainer" containerID="966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9"
Jul 01 19:42:31.927201 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:31.927174    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9\": container with ID starting with 966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9 not found: ID does not exist" containerID="966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9"
Jul 01 19:42:31.927273 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:31.927209    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9"} err="failed to get container status \"966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9\": rpc error: code = NotFound desc = could not find container \"966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9\": container with ID starting with 966705835f41bc150835301590b119a5b99d5bf92c4d5b2ea5b6d01eae53aba9 not found: ID does not exist"
Jul 01 19:42:32.036345 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.036322    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a910e61-15e1-46c0-82da-d2781902c582-kube-rbac-proxy-sar-config\") pod \"8a910e61-15e1-46c0-82da-d2781902c582\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") "
Jul 01 19:42:32.036506 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.036369    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a910e61-15e1-46c0-82da-d2781902c582-proxy-tls\") pod \"8a910e61-15e1-46c0-82da-d2781902c582\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") "
Jul 01 19:42:32.036506 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.036387    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9btbr\" (UniqueName: \"kubernetes.io/projected/8a910e61-15e1-46c0-82da-d2781902c582-kube-api-access-9btbr\") pod \"8a910e61-15e1-46c0-82da-d2781902c582\" (UID: \"8a910e61-15e1-46c0-82da-d2781902c582\") "
Jul 01 19:42:32.036748 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.036726    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a910e61-15e1-46c0-82da-d2781902c582-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a910e61-15e1-46c0-82da-d2781902c582" (UID: "8a910e61-15e1-46c0-82da-d2781902c582"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:32.038422 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.038395    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a910e61-15e1-46c0-82da-d2781902c582-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a910e61-15e1-46c0-82da-d2781902c582" (UID: "8a910e61-15e1-46c0-82da-d2781902c582"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:32.038540 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.038474    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a910e61-15e1-46c0-82da-d2781902c582-kube-api-access-9btbr" (OuterVolumeSpecName: "kube-api-access-9btbr") pod "8a910e61-15e1-46c0-82da-d2781902c582" (UID: "8a910e61-15e1-46c0-82da-d2781902c582"). InnerVolumeSpecName "kube-api-access-9btbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:42:32.137183 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.137145    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a910e61-15e1-46c0-82da-d2781902c582-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:42:32.137183 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.137182    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a910e61-15e1-46c0-82da-d2781902c582-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:42:32.137183 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.137191    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btbr\" (UniqueName: \"kubernetes.io/projected/8a910e61-15e1-46c0-82da-d2781902c582-kube-api-access-9btbr\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:42:32.233153 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.233127    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"]
Jul 01 19:42:32.236433 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.236411    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d109c-predictor-5d9b9bf885-t4frr"]
Jul 01 19:42:32.558298 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.558274    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:42:32.742501 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.742413    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46fc9c-d916-4629-9fce-f865f714f516-kube-rbac-proxy-sar-config\") pod \"aa46fc9c-d916-4629-9fce-f865f714f516\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") "
Jul 01 19:42:32.742635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.742529    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46fc9c-d916-4629-9fce-f865f714f516-proxy-tls\") pod \"aa46fc9c-d916-4629-9fce-f865f714f516\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") "
Jul 01 19:42:32.742635 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.742551    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl6zs\" (UniqueName: \"kubernetes.io/projected/aa46fc9c-d916-4629-9fce-f865f714f516-kube-api-access-hl6zs\") pod \"aa46fc9c-d916-4629-9fce-f865f714f516\" (UID: \"aa46fc9c-d916-4629-9fce-f865f714f516\") "
Jul 01 19:42:32.742857 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.742824    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa46fc9c-d916-4629-9fce-f865f714f516-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa46fc9c-d916-4629-9fce-f865f714f516" (UID: "aa46fc9c-d916-4629-9fce-f865f714f516"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:42:32.744718 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.744692    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa46fc9c-d916-4629-9fce-f865f714f516-kube-api-access-hl6zs" (OuterVolumeSpecName: "kube-api-access-hl6zs") pod "aa46fc9c-d916-4629-9fce-f865f714f516" (UID: "aa46fc9c-d916-4629-9fce-f865f714f516"). InnerVolumeSpecName "kube-api-access-hl6zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:42:32.744718 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.744704    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa46fc9c-d916-4629-9fce-f865f714f516-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa46fc9c-d916-4629-9fce-f865f714f516" (UID: "aa46fc9c-d916-4629-9fce-f865f714f516"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:42:32.843413 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.843388    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46fc9c-d916-4629-9fce-f865f714f516-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:42:32.843413 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.843410    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl6zs\" (UniqueName: \"kubernetes.io/projected/aa46fc9c-d916-4629-9fce-f865f714f516-kube-api-access-hl6zs\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:42:32.843555 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.843420    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46fc9c-d916-4629-9fce-f865f714f516-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:42:32.916471 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.916426    2576 generic.go:356] "Generic (PLEG): container finished" podID="aa46fc9c-d916-4629-9fce-f865f714f516" containerID="fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa" exitCode=0
Jul 01 19:42:32.916763 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.916477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" event={"ID":"aa46fc9c-d916-4629-9fce-f865f714f516","Type":"ContainerDied","Data":"fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa"}
Jul 01 19:42:32.916763 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.916499    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"
Jul 01 19:42:32.916763 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.916535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b" event={"ID":"aa46fc9c-d916-4629-9fce-f865f714f516","Type":"ContainerDied","Data":"71f3cc9bc353a90611d1af8ea99c38beea942d165f63c1c6172a0886e591f7c6"}
Jul 01 19:42:32.916763 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.916561    2576 scope.go:117] "RemoveContainer" containerID="0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3"
Jul 01 19:42:32.917017 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.916981    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 01 19:42:32.925842 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.925825    2576 scope.go:117] "RemoveContainer" containerID="fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa"
Jul 01 19:42:32.933233 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.933214    2576 scope.go:117] "RemoveContainer" containerID="0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3"
Jul 01 19:42:32.933471 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:32.933438    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3\": container with ID starting with 0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3 not found: ID does not exist" containerID="0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3"
Jul 01 19:42:32.933559 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.933478    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3"} err="failed to get container status \"0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3\": rpc error: code = NotFound desc = could not find container \"0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3\": container with ID starting with 0319a8ae7dc2ccfea5692844dc58d3182416f6cdb2d737ad7eba4ccc5b7ec3b3 not found: ID does not exist"
Jul 01 19:42:32.933559 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.933495    2576 scope.go:117] "RemoveContainer" containerID="fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa"
Jul 01 19:42:32.933720 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:32.933698    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa\": container with ID starting with fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa not found: ID does not exist" containerID="fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa"
Jul 01 19:42:32.933778 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.933731    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa"} err="failed to get container status \"fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa\": rpc error: code = NotFound desc = could not find container \"fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa\": container with ID starting with fdaff3a912ceb4f2ed427bf0679f52de7051c816851d8b9f1bb865061d1f4dfa not found: ID does not exist"
Jul 01 19:42:32.941612 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.941588    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"]
Jul 01 19:42:32.945158 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:32.945137    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d109c-predictor-6dc7dbbb44-vgv9b"]
Jul 01 19:42:33.751658 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:33.751625    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a910e61-15e1-46c0-82da-d2781902c582" path="/var/lib/kubelet/pods/8a910e61-15e1-46c0-82da-d2781902c582/volumes"
Jul 01 19:42:33.752061 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:33.752049    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" path="/var/lib/kubelet/pods/aa46fc9c-d916-4629-9fce-f865f714f516/volumes"
Jul 01 19:42:35.696168 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:35.696139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:42:36.914109 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:36.914077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:42:36.914566 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:36.914536    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 01 19:42:37.706128 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:37.706091    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:42:37.921728 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:37.921701    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:42:37.922162 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:37.922124    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 01 19:42:46.915199 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:46.915162    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 01 19:42:47.922657 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:47.922613    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 01 19:42:56.914942 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:56.914902    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 01 19:42:57.922313 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:57.922274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 01 19:42:59.073583 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.073543    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"]
Jul 01 19:42:59.073979 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.073785    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container" containerID="cri-o://2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a" gracePeriod=30
Jul 01 19:42:59.073979 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.073836    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kube-rbac-proxy" containerID="cri-o://1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca" gracePeriod=30
Jul 01 19:42:59.098582 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.098558    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"]
Jul 01 19:42:59.099089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099062    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kube-rbac-proxy"
Jul 01 19:42:59.099089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099083    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kube-rbac-proxy"
Jul 01 19:42:59.099240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099102    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kube-rbac-proxy"
Jul 01 19:42:59.099240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099109    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kube-rbac-proxy"
Jul 01 19:42:59.099240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099135    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container"
Jul 01 19:42:59.099240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099141    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container"
Jul 01 19:42:59.099240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099152    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container"
Jul 01 19:42:59.099240 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099157    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container"
Jul 01 19:42:59.099502 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099328    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kube-rbac-proxy"
Jul 01 19:42:59.099502 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099369    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kserve-container"
Jul 01 19:42:59.099502 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099386    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46fc9c-d916-4629-9fce-f865f714f516" containerName="kserve-container"
Jul 01 19:42:59.099502 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.099393    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a910e61-15e1-46c0-82da-d2781902c582" containerName="kube-rbac-proxy"
Jul 01 19:42:59.102987 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.102967    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.105742 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.105722    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-25c73-predictor-serving-cert\""
Jul 01 19:42:59.105855 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.105777    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-25c73-kube-rbac-proxy-sar-config\""
Jul 01 19:42:59.116644 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.116172    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"]
Jul 01 19:42:59.165283 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.165253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.165413 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.165316    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.165413 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.165370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b8x\" (UniqueName: \"kubernetes.io/projected/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-api-access-t8b8x\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.192669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.192635    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"]
Jul 01 19:42:59.201769 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.201750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.204375 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.204357    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-25c73-kube-rbac-proxy-sar-config\""
Jul 01 19:42:59.204375 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.204366    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-25c73-predictor-serving-cert\""
Jul 01 19:42:59.210356 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.210332    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"]
Jul 01 19:42:59.236340 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.236303    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"]
Jul 01 19:42:59.237036 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.237004    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kube-rbac-proxy" containerID="cri-o://74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8" gracePeriod=30
Jul 01 19:42:59.238313 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.238258    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container" containerID="cri-o://99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248" gracePeriod=30
Jul 01 19:42:59.267484 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.267425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.267649 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.267493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.267649 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.267540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpz8g\" (UniqueName: \"kubernetes.io/projected/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-api-access-tpz8g\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.267649 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.267578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97ab1d3-1f52-4493-87ec-fcc966e68abc-proxy-tls\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.267649 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:59.267581    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-25c73-predictor-serving-cert: secret "success-200-isvc-25c73-predictor-serving-cert" not found
Jul 01 19:42:59.267649 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:59.267652    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls podName:b8128d6a-7d30-43d4-b37b-2cb6d3256c13 nodeName:}" failed. No retries permitted until 2026-07-01 19:42:59.767629353 +0000 UTC m=+1306.527156052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls") pod "success-200-isvc-25c73-predictor-746989f766-6kkr9" (UID: "b8128d6a-7d30-43d4-b37b-2cb6d3256c13") : secret "success-200-isvc-25c73-predictor-serving-cert" not found
Jul 01 19:42:59.268124 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.267676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b8x\" (UniqueName: \"kubernetes.io/projected/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-api-access-t8b8x\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.268124 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.267718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.268509 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.268486    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.276626 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.276608    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b8x\" (UniqueName: \"kubernetes.io/projected/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-api-access-t8b8x\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.369028 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.369003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97ab1d3-1f52-4493-87ec-fcc966e68abc-proxy-tls\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.369127 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.369102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.369170 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.369129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpz8g\" (UniqueName: \"kubernetes.io/projected/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-api-access-tpz8g\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.369702 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.369680    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.371247 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.371224    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97ab1d3-1f52-4493-87ec-fcc966e68abc-proxy-tls\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.379470 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.379434    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpz8g\" (UniqueName: \"kubernetes.io/projected/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-api-access-tpz8g\") pod \"error-404-isvc-25c73-predictor-6ddf4c5487-md7nf\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") " pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.513824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.513796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:42:59.649215 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.649185    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"]
Jul 01 19:42:59.650631 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:42:59.650601    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97ab1d3_1f52_4493_87ec_fcc966e68abc.slice/crio-bd5df3b25d4e8f757ee1e09e1163212768a35f1ebe6a122c094ef65f7b36566d WatchSource:0}: Error finding container bd5df3b25d4e8f757ee1e09e1163212768a35f1ebe6a122c094ef65f7b36566d: Status 404 returned error can't find the container with id bd5df3b25d4e8f757ee1e09e1163212768a35f1ebe6a122c094ef65f7b36566d
Jul 01 19:42:59.772834 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:42:59.772812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:42:59.772968 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:59.772950    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-25c73-predictor-serving-cert: secret "success-200-isvc-25c73-predictor-serving-cert" not found
Jul 01 19:42:59.773031 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:42:59.773020    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls podName:b8128d6a-7d30-43d4-b37b-2cb6d3256c13 nodeName:}" failed. No retries permitted until 2026-07-01 19:43:00.77299746 +0000 UTC m=+1307.532524159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls") pod "success-200-isvc-25c73-predictor-746989f766-6kkr9" (UID: "b8128d6a-7d30-43d4-b37b-2cb6d3256c13") : secret "success-200-isvc-25c73-predictor-serving-cert" not found
Jul 01 19:43:00.057090 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.057050    2576 generic.go:356] "Generic (PLEG): container finished" podID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerID="1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca" exitCode=2
Jul 01 19:43:00.057243 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.057110    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" event={"ID":"c217f2fc-cd4e-4286-a5fd-961a1809ca4f","Type":"ContainerDied","Data":"1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca"}
Jul 01 19:43:00.059285 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.059263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" event={"ID":"f97ab1d3-1f52-4493-87ec-fcc966e68abc","Type":"ContainerStarted","Data":"ae9c6a03f382b4ecd819d9fed8b7c9a0aeb84570a15b32e42f253d82d6ec602b"}
Jul 01 19:43:00.059403 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.059314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" event={"ID":"f97ab1d3-1f52-4493-87ec-fcc966e68abc","Type":"ContainerStarted","Data":"9f349e5ea0ee9132a48c3e4cc5d801b0ea432b1cdacc7b81ab91374f147b2d32"}
Jul 01 19:43:00.059403 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.059337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" event={"ID":"f97ab1d3-1f52-4493-87ec-fcc966e68abc","Type":"ContainerStarted","Data":"bd5df3b25d4e8f757ee1e09e1163212768a35f1ebe6a122c094ef65f7b36566d"}
Jul 01 19:43:00.059548 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.059487    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:43:00.061149 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.061127    2576 generic.go:356] "Generic (PLEG): container finished" podID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerID="74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8" exitCode=2
Jul 01 19:43:00.061255 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.061200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" event={"ID":"c3ca7119-e603-4b2d-93f8-2eea92ef0800","Type":"ContainerDied","Data":"74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8"}
Jul 01 19:43:00.077258 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.077211    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podStartSLOduration=1.077197494 podStartE2EDuration="1.077197494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:00.076253854 +0000 UTC m=+1306.835780586" watchObservedRunningTime="2026-07-01 19:43:00.077197494 +0000 UTC m=+1306.836724215"
Jul 01 19:43:00.690130 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.690089    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 01 19:43:00.780880 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.780844    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:43:00.783164 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.783138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls\") pod \"success-200-isvc-25c73-predictor-746989f766-6kkr9\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") " pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:43:00.914764 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:00.914733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:43:01.052469 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:01.052432    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"]
Jul 01 19:43:01.054164 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:43:01.054135    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8128d6a_7d30_43d4_b37b_2cb6d3256c13.slice/crio-47c100e1fe6eff12ac493ceb9cd83c009fc8b520d192ca95ce0397eab8391181 WatchSource:0}: Error finding container 47c100e1fe6eff12ac493ceb9cd83c009fc8b520d192ca95ce0397eab8391181: Status 404 returned error can't find the container with id 47c100e1fe6eff12ac493ceb9cd83c009fc8b520d192ca95ce0397eab8391181
Jul 01 19:43:01.068021 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:01.067987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" event={"ID":"b8128d6a-7d30-43d4-b37b-2cb6d3256c13","Type":"ContainerStarted","Data":"47c100e1fe6eff12ac493ceb9cd83c009fc8b520d192ca95ce0397eab8391181"}
Jul 01 19:43:01.068326 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:01.068309    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:43:01.069824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:01.069801    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 01 19:43:02.075388 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.075344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" event={"ID":"b8128d6a-7d30-43d4-b37b-2cb6d3256c13","Type":"ContainerStarted","Data":"1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a"}
Jul 01 19:43:02.075796 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.075417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" event={"ID":"b8128d6a-7d30-43d4-b37b-2cb6d3256c13","Type":"ContainerStarted","Data":"61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa"}
Jul 01 19:43:02.075796 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.075651    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 01 19:43:02.095608 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.095561    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podStartSLOduration=3.095547141 podStartE2EDuration="3.095547141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:02.093720205 +0000 UTC m=+1308.853246924" watchObservedRunningTime="2026-07-01 19:43:02.095547141 +0000 UTC m=+1308.855073862"
Jul 01 19:43:02.416735 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.416710    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:43:02.496727 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.496691    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-rbac-proxy-sar-config\") pod \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") "
Jul 01 19:43:02.496902 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.496752    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vgsc\" (UniqueName: \"kubernetes.io/projected/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-api-access-4vgsc\") pod \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") "
Jul 01 19:43:02.496902 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.496798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ca7119-e603-4b2d-93f8-2eea92ef0800-proxy-tls\") pod \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\" (UID: \"c3ca7119-e603-4b2d-93f8-2eea92ef0800\") "
Jul 01 19:43:02.497061 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.497034    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3ca7119-e603-4b2d-93f8-2eea92ef0800" (UID: "c3ca7119-e603-4b2d-93f8-2eea92ef0800"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:43:02.498874 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.498840    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3ca7119-e603-4b2d-93f8-2eea92ef0800-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3ca7119-e603-4b2d-93f8-2eea92ef0800" (UID: "c3ca7119-e603-4b2d-93f8-2eea92ef0800"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:43:02.498989 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.498893    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-api-access-4vgsc" (OuterVolumeSpecName: "kube-api-access-4vgsc") pod "c3ca7119-e603-4b2d-93f8-2eea92ef0800" (UID: "c3ca7119-e603-4b2d-93f8-2eea92ef0800"). InnerVolumeSpecName "kube-api-access-4vgsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:43:02.598277 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.598243    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vgsc\" (UniqueName: \"kubernetes.io/projected/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-api-access-4vgsc\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:02.598277 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.598272    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3ca7119-e603-4b2d-93f8-2eea92ef0800-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:02.598494 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.598284    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3ca7119-e603-4b2d-93f8-2eea92ef0800-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:02.701670 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.701582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 01 19:43:02.909252 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:02.909225    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:43:03.002224 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.002156    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sh2w\" (UniqueName: \"kubernetes.io/projected/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-api-access-9sh2w\") pod \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") "
Jul 01 19:43:03.002224 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.002204    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-rbac-proxy-sar-config\") pod \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") "
Jul 01 19:43:03.002405 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.002276    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls\") pod \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\" (UID: \"c217f2fc-cd4e-4286-a5fd-961a1809ca4f\") "
Jul 01 19:43:03.002649 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.002618    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c217f2fc-cd4e-4286-a5fd-961a1809ca4f" (UID: "c217f2fc-cd4e-4286-a5fd-961a1809ca4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:43:03.004194 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.004172    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c217f2fc-cd4e-4286-a5fd-961a1809ca4f" (UID: "c217f2fc-cd4e-4286-a5fd-961a1809ca4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:43:03.004266 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.004176    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-api-access-9sh2w" (OuterVolumeSpecName: "kube-api-access-9sh2w") pod "c217f2fc-cd4e-4286-a5fd-961a1809ca4f" (UID: "c217f2fc-cd4e-4286-a5fd-961a1809ca4f"). InnerVolumeSpecName "kube-api-access-9sh2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:43:03.081227 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.081200    2576 generic.go:356] "Generic (PLEG): container finished" podID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerID="99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248" exitCode=0
Jul 01 19:43:03.081623 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.081249    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"
Jul 01 19:43:03.081623 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.081294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" event={"ID":"c3ca7119-e603-4b2d-93f8-2eea92ef0800","Type":"ContainerDied","Data":"99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248"}
Jul 01 19:43:03.081623 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.081364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj" event={"ID":"c3ca7119-e603-4b2d-93f8-2eea92ef0800","Type":"ContainerDied","Data":"94063cde7a9fc4a88c9efae40a8a5037502a32ce7f42dcea4071401617ae9818"}
Jul 01 19:43:03.081623 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.081406    2576 scope.go:117] "RemoveContainer" containerID="74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8"
Jul 01 19:43:03.083032 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.083010    2576 generic.go:356] "Generic (PLEG): container finished" podID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerID="2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a" exitCode=0
Jul 01 19:43:03.083144 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.083055    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"
Jul 01 19:43:03.083144 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.083059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" event={"ID":"c217f2fc-cd4e-4286-a5fd-961a1809ca4f","Type":"ContainerDied","Data":"2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a"}
Jul 01 19:43:03.083144 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.083120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl" event={"ID":"c217f2fc-cd4e-4286-a5fd-961a1809ca4f","Type":"ContainerDied","Data":"e19487c9e8c0a9b9194a7b77137a3193e4824e59635937d74afb72b0b94826bb"}
Jul 01 19:43:03.083507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.083482    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:43:03.083616 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.083538    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:43:03.085261 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.085235    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 01 19:43:03.103107 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.103088    2576 scope.go:117] "RemoveContainer" containerID="99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248"
Jul 01 19:43:03.103216 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.103194    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:03.103273 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.103224    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sh2w\" (UniqueName: \"kubernetes.io/projected/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-api-access-9sh2w\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:03.103273 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.103239    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c217f2fc-cd4e-4286-a5fd-961a1809ca4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:03.110866 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.110844    2576 scope.go:117] "RemoveContainer" containerID="74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8"
Jul 01 19:43:03.111329 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:03.111308    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8\": container with ID starting with 74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8 not found: ID does not exist" containerID="74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8"
Jul 01 19:43:03.111410 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.111333    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8"} err="failed to get container status \"74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8\": rpc error: code = NotFound desc = could not find container \"74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8\": container with ID starting with 74c207313180b1baba95163a6bd4b957f12cbca5e6fd196b4e1c4105944fbdf8 not found: ID does not exist"
Jul 01 19:43:03.111410 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.111348    2576 scope.go:117] "RemoveContainer" containerID="99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248"
Jul 01 19:43:03.111624 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:03.111608    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248\": container with ID starting with 99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248 not found: ID does not exist" containerID="99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248"
Jul 01 19:43:03.111672 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.111629    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248"} err="failed to get container status \"99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248\": rpc error: code = NotFound desc = could not find container \"99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248\": container with ID starting with 99593e8ff12f7ac439451618a44db1c0f513d227f8b4d72c268853fb31dc3248 not found: ID does not exist"
Jul 01 19:43:03.111672 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.111643    2576 scope.go:117] "RemoveContainer" containerID="1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca"
Jul 01 19:43:03.113068 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.113048    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"]
Jul 01 19:43:03.117101 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.117082    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-44b6c-predictor-65b6d49b65-pmtmj"]
Jul 01 19:43:03.119322 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.119302    2576 scope.go:117] "RemoveContainer" containerID="2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a"
Jul 01 19:43:03.126412 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.126388    2576 scope.go:117] "RemoveContainer" containerID="1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca"
Jul 01 19:43:03.126669 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:03.126652    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca\": container with ID starting with 1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca not found: ID does not exist" containerID="1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca"
Jul 01 19:43:03.126728 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.126673    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca"} err="failed to get container status \"1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca\": rpc error: code = NotFound desc = could not find container \"1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca\": container with ID starting with 1746c86d7d8e9869d2fe6ecf3fab712fd78c148352ad66b538eb0edb626325ca not found: ID does not exist"
Jul 01 19:43:03.126728 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.126687    2576 scope.go:117] "RemoveContainer" containerID="2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a"
Jul 01 19:43:03.126933 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:03.126914    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a\": container with ID starting with 2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a not found: ID does not exist" containerID="2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a"
Jul 01 19:43:03.126986 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.126936    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a"} err="failed to get container status \"2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a\": rpc error: code = NotFound desc = could not find container \"2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a\": container with ID starting with 2cd5dcc4ba20b0185f0758ab2ab8d7518466d323266bca1b481c41465cd81a2a not found: ID does not exist"
Jul 01 19:43:03.135404 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.135385    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"]
Jul 01 19:43:03.149289 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.149260    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44b6c-predictor-788df44c68-prgzl"]
Jul 01 19:43:03.752628 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.752593    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" path="/var/lib/kubelet/pods/c217f2fc-cd4e-4286-a5fd-961a1809ca4f/volumes"
Jul 01 19:43:03.753060 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:03.753045    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" path="/var/lib/kubelet/pods/c3ca7119-e603-4b2d-93f8-2eea92ef0800/volumes"
Jul 01 19:43:04.089257 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:04.089216    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 01 19:43:06.914995 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:06.914959    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 01 19:43:07.080688 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:07.080658    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:43:07.081137 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:07.081109    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 01 19:43:07.922290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:07.922253    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 01 19:43:09.094641 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:09.094610    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:43:09.095078 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:09.095051    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 01 19:43:16.915976 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:16.915941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:43:17.081317 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:17.081275    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 01 19:43:17.922799 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:17.922769    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:43:19.095203 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:19.095160    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 01 19:43:27.081661 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:27.081616    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 01 19:43:29.095973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:29.095932    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 01 19:43:37.082092 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:37.082007    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 01 19:43:39.044444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.044412    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"]
Jul 01 19:43:39.044903 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.044721    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container" containerID="cri-o://dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626" gracePeriod=30
Jul 01 19:43:39.045258 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.044975    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kube-rbac-proxy" containerID="cri-o://6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4" gracePeriod=30
Jul 01 19:43:39.095377 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.095340    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Jul 01 19:43:39.108319 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.108295    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"]
Jul 01 19:43:39.108535 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.108511    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container" containerID="cri-o://3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11" gracePeriod=30
Jul 01 19:43:39.108589 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.108541    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kube-rbac-proxy" containerID="cri-o://1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33" gracePeriod=30
Jul 01 19:43:39.113747 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.113714    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"]
Jul 01 19:43:39.114211 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114195    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container"
Jul 01 19:43:39.114279 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114213    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container"
Jul 01 19:43:39.114279 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114225    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kube-rbac-proxy"
Jul 01 19:43:39.114279 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114231    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kube-rbac-proxy"
Jul 01 19:43:39.114279 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114252    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container"
Jul 01 19:43:39.114279 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114258    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container"
Jul 01 19:43:39.114444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114282    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kube-rbac-proxy"
Jul 01 19:43:39.114444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114288    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kube-rbac-proxy"
Jul 01 19:43:39.114444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114367    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kube-rbac-proxy"
Jul 01 19:43:39.114444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114379    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ca7119-e603-4b2d-93f8-2eea92ef0800" containerName="kserve-container"
Jul 01 19:43:39.114444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114387    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kube-rbac-proxy"
Jul 01 19:43:39.114444 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.114397    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c217f2fc-cd4e-4286-a5fd-961a1809ca4f" containerName="kserve-container"
Jul 01 19:43:39.119528 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.119512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.123849 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.123832    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f705b-kube-rbac-proxy-sar-config\""
Jul 01 19:43:39.123923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.123833    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f705b-predictor-serving-cert\""
Jul 01 19:43:39.129969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.129943    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"]
Jul 01 19:43:39.203818 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.203788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nf7v\" (UniqueName: \"kubernetes.io/projected/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-api-access-4nf7v\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.203981 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.203832    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-proxy-tls\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.203981 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.203860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.226235 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.226213    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"]
Jul 01 19:43:39.229966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.229950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.236194 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.236043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f705b-predictor-serving-cert\""
Jul 01 19:43:39.236194 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.236056    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f705b-kube-rbac-proxy-sar-config\""
Jul 01 19:43:39.245110 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.245086    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"]
Jul 01 19:43:39.264122 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.264093    2576 generic.go:356] "Generic (PLEG): container finished" podID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerID="6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4" exitCode=2
Jul 01 19:43:39.264256 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.264161    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" event={"ID":"492f1bb4-3f05-44d8-bb47-1a1b044cf510","Type":"ContainerDied","Data":"6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4"}
Jul 01 19:43:39.266097 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.266066    2576 generic.go:356] "Generic (PLEG): container finished" podID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerID="1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33" exitCode=2
Jul 01 19:43:39.266223 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.266128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" event={"ID":"334395e0-b09e-41b9-bb5c-4eb4b371eaa8","Type":"ContainerDied","Data":"1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33"}
Jul 01 19:43:39.305217 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.305156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nf7v\" (UniqueName: \"kubernetes.io/projected/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-api-access-4nf7v\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.305217 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.305188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlbh\" (UniqueName: \"kubernetes.io/projected/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-api-access-bxlbh\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.305217 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.305210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-proxy-tls\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.305392 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.305373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-proxy-tls\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.305429 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.305394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.305497 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.305434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.306044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.306019    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.307871 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.307851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-proxy-tls\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.316328 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.316304    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nf7v\" (UniqueName: \"kubernetes.io/projected/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-api-access-4nf7v\") pod \"success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") " pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.406222 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.406191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.406404 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.406263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlbh\" (UniqueName: \"kubernetes.io/projected/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-api-access-bxlbh\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.406404 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.406286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-proxy-tls\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.406926 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.406899    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.408706 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.408683    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-proxy-tls\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.415250 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.415228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlbh\" (UniqueName: \"kubernetes.io/projected/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-api-access-bxlbh\") pod \"error-404-isvc-f705b-predictor-d6ff57dff-d2dhp\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") " pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.431997 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.431978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:39.549475 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.539973    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:39.571163 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.571136    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"]
Jul 01 19:43:39.573043 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:43:39.573012    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a79a16d_57d1_4d3c_ae44_fa833c56ef89.slice/crio-645637b51db165c4adcea350f9811a5ea17a496d9ae5353f75977131d1456516 WatchSource:0}: Error finding container 645637b51db165c4adcea350f9811a5ea17a496d9ae5353f75977131d1456516: Status 404 returned error can't find the container with id 645637b51db165c4adcea350f9811a5ea17a496d9ae5353f75977131d1456516
Jul 01 19:43:39.678367 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:39.678344    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"]
Jul 01 19:43:39.689492 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:43:39.689469    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e25b7d_d3e1_47cd_a289_1e3df644af1c.slice/crio-39f91762f0fc41b457950321457b85bbd818275f3640d52c3a8f84ce717c3b2c WatchSource:0}: Error finding container 39f91762f0fc41b457950321457b85bbd818275f3640d52c3a8f84ce717c3b2c: Status 404 returned error can't find the container with id 39f91762f0fc41b457950321457b85bbd818275f3640d52c3a8f84ce717c3b2c
Jul 01 19:43:40.272498 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.272460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" event={"ID":"f7e25b7d-d3e1-47cd-a289-1e3df644af1c","Type":"ContainerStarted","Data":"b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944"}
Jul 01 19:43:40.272923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.272526    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:40.272923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.272554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" event={"ID":"f7e25b7d-d3e1-47cd-a289-1e3df644af1c","Type":"ContainerStarted","Data":"018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907"}
Jul 01 19:43:40.272923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.272584    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" event={"ID":"f7e25b7d-d3e1-47cd-a289-1e3df644af1c","Type":"ContainerStarted","Data":"39f91762f0fc41b457950321457b85bbd818275f3640d52c3a8f84ce717c3b2c"}
Jul 01 19:43:40.272923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.272612    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:40.274485 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.274436    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" event={"ID":"3a79a16d-57d1-4d3c-ae44-fa833c56ef89","Type":"ContainerStarted","Data":"fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea"}
Jul 01 19:43:40.274485 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.274463    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 01 19:43:40.274667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.274515    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" event={"ID":"3a79a16d-57d1-4d3c-ae44-fa833c56ef89","Type":"ContainerStarted","Data":"7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b"}
Jul 01 19:43:40.274667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.274550    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:40.274667 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.274573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" event={"ID":"3a79a16d-57d1-4d3c-ae44-fa833c56ef89","Type":"ContainerStarted","Data":"645637b51db165c4adcea350f9811a5ea17a496d9ae5353f75977131d1456516"}
Jul 01 19:43:40.321119 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.321075    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podStartSLOduration=1.321061799 podStartE2EDuration="1.321061799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:40.319662929 +0000 UTC m=+1347.079189661" watchObservedRunningTime="2026-07-01 19:43:40.321061799 +0000 UTC m=+1347.080588519"
Jul 01 19:43:40.322037 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:40.322006    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podStartSLOduration=1.321998118 podStartE2EDuration="1.321998118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:43:40.295815591 +0000 UTC m=+1347.055342312" watchObservedRunningTime="2026-07-01 19:43:40.321998118 +0000 UTC m=+1347.081524838"
Jul 01 19:43:41.279601 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:41.279560    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 01 19:43:41.280048 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:41.279660    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:41.281082 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:41.281054    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 01 19:43:41.911028 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:41.910981    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 01 19:43:42.284207 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.284165    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 01 19:43:42.818559 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.818511    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:43:42.866539 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.866514    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:43:42.932740 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.932679    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqdw9\" (UniqueName: \"kubernetes.io/projected/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-api-access-jqdw9\") pod \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") "
Jul 01 19:43:42.932740 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.932715    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-proxy-tls\") pod \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") "
Jul 01 19:43:42.932740 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.932736    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-rbac-proxy-sar-config\") pod \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\" (UID: \"334395e0-b09e-41b9-bb5c-4eb4b371eaa8\") "
Jul 01 19:43:42.932973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.932784    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls\") pod \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") "
Jul 01 19:43:42.932973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.932848    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-rbac-proxy-sar-config\") pod \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") "
Jul 01 19:43:42.932973 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.932908    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97x5j\" (UniqueName: \"kubernetes.io/projected/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-api-access-97x5j\") pod \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\" (UID: \"492f1bb4-3f05-44d8-bb47-1a1b044cf510\") "
Jul 01 19:43:42.933196 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.933166    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "334395e0-b09e-41b9-bb5c-4eb4b371eaa8" (UID: "334395e0-b09e-41b9-bb5c-4eb4b371eaa8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:43:42.933334 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.933240    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "492f1bb4-3f05-44d8-bb47-1a1b044cf510" (UID: "492f1bb4-3f05-44d8-bb47-1a1b044cf510"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:43:42.934794 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.934774    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "334395e0-b09e-41b9-bb5c-4eb4b371eaa8" (UID: "334395e0-b09e-41b9-bb5c-4eb4b371eaa8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:43:42.934900 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.934869    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-api-access-97x5j" (OuterVolumeSpecName: "kube-api-access-97x5j") pod "492f1bb4-3f05-44d8-bb47-1a1b044cf510" (UID: "492f1bb4-3f05-44d8-bb47-1a1b044cf510"). InnerVolumeSpecName "kube-api-access-97x5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:43:42.935202 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.935186    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "492f1bb4-3f05-44d8-bb47-1a1b044cf510" (UID: "492f1bb4-3f05-44d8-bb47-1a1b044cf510"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:43:42.935202 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:42.935190    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-api-access-jqdw9" (OuterVolumeSpecName: "kube-api-access-jqdw9") pod "334395e0-b09e-41b9-bb5c-4eb4b371eaa8" (UID: "334395e0-b09e-41b9-bb5c-4eb4b371eaa8"). InnerVolumeSpecName "kube-api-access-jqdw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:43:43.033970 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.033941    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:43.033970 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.033962    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97x5j\" (UniqueName: \"kubernetes.io/projected/492f1bb4-3f05-44d8-bb47-1a1b044cf510-kube-api-access-97x5j\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:43.033970 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.033973    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqdw9\" (UniqueName: \"kubernetes.io/projected/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-api-access-jqdw9\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:43.034159 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.033982    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:43.034159 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.033992    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/334395e0-b09e-41b9-bb5c-4eb4b371eaa8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:43.034159 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.034000    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/492f1bb4-3f05-44d8-bb47-1a1b044cf510-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:43:43.290039 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.290009    2576 generic.go:356] "Generic (PLEG): container finished" podID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerID="dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626" exitCode=0
Jul 01 19:43:43.290490 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.290073    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"
Jul 01 19:43:43.290490 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.290104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" event={"ID":"492f1bb4-3f05-44d8-bb47-1a1b044cf510","Type":"ContainerDied","Data":"dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626"}
Jul 01 19:43:43.290490 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.290173    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57" event={"ID":"492f1bb4-3f05-44d8-bb47-1a1b044cf510","Type":"ContainerDied","Data":"def9737aa38991faa1b62c571a6d83d64caf0e8b671e33c444f89730bc11ff1d"}
Jul 01 19:43:43.290490 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.290210    2576 scope.go:117] "RemoveContainer" containerID="6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4"
Jul 01 19:43:43.291835 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.291812    2576 generic.go:356] "Generic (PLEG): container finished" podID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerID="3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11" exitCode=0
Jul 01 19:43:43.292012 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.291855    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"
Jul 01 19:43:43.292012 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.291897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" event={"ID":"334395e0-b09e-41b9-bb5c-4eb4b371eaa8","Type":"ContainerDied","Data":"3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11"}
Jul 01 19:43:43.292012 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.291959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px" event={"ID":"334395e0-b09e-41b9-bb5c-4eb4b371eaa8","Type":"ContainerDied","Data":"1dc28690af091ecfa9639d3a482d1bcb4cbeff191af7f9e9a9bef5494f967758"}
Jul 01 19:43:43.299053 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.299030    2576 scope.go:117] "RemoveContainer" containerID="dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626"
Jul 01 19:43:43.307262 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.307246    2576 scope.go:117] "RemoveContainer" containerID="6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4"
Jul 01 19:43:43.307510 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:43.307483    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4\": container with ID starting with 6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4 not found: ID does not exist" containerID="6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4"
Jul 01 19:43:43.307599 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.307522    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4"} err="failed to get container status \"6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4\": rpc error: code = NotFound desc = could not find container \"6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4\": container with ID starting with 6765fdb394de89189a0f1a6ecf8ecc50fcbcf7e6f0f913ec2f630cd8c905a0d4 not found: ID does not exist"
Jul 01 19:43:43.307599 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.307545    2576 scope.go:117] "RemoveContainer" containerID="dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626"
Jul 01 19:43:43.307792 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:43.307777    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626\": container with ID starting with dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626 not found: ID does not exist" containerID="dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626"
Jul 01 19:43:43.307852 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.307801    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626"} err="failed to get container status \"dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626\": rpc error: code = NotFound desc = could not find container \"dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626\": container with ID starting with dfac9a4b81e9e1fa60ddc2b1503f704cdf647bb52c765b3c62da2d21d9675626 not found: ID does not exist"
Jul 01 19:43:43.307852 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.307821    2576 scope.go:117] "RemoveContainer" containerID="1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33"
Jul 01 19:43:43.315068 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.315051    2576 scope.go:117] "RemoveContainer" containerID="3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11"
Jul 01 19:43:43.317647 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.317607    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"]
Jul 01 19:43:43.319809 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.319788    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7f91-predictor-69c44d6649-89l57"]
Jul 01 19:43:43.322891 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.322878    2576 scope.go:117] "RemoveContainer" containerID="1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33"
Jul 01 19:43:43.323117 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:43.323103    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33\": container with ID starting with 1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33 not found: ID does not exist" containerID="1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33"
Jul 01 19:43:43.323160 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.323122    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33"} err="failed to get container status \"1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33\": rpc error: code = NotFound desc = could not find container \"1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33\": container with ID starting with 1f7d7eeae2e18af8edc88e80b0fcb8188624f658ef9832d473836aa06c65bc33 not found: ID does not exist"
Jul 01 19:43:43.323160 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.323136    2576 scope.go:117] "RemoveContainer" containerID="3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11"
Jul 01 19:43:43.323360 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:43:43.323343    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11\": container with ID starting with 3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11 not found: ID does not exist" containerID="3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11"
Jul 01 19:43:43.323408 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.323365    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11"} err="failed to get container status \"3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11\": rpc error: code = NotFound desc = could not find container \"3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11\": container with ID starting with 3c0ad80e02a40f1e3256dbc175cb2323b4fa7a9bb39062fc77c1921c9c383f11 not found: ID does not exist"
Jul 01 19:43:43.336063 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.336018    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"]
Jul 01 19:43:43.338356 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.338336    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7f91-predictor-5974f76cdd-qs7px"]
Jul 01 19:43:43.752557 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.752525    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" path="/var/lib/kubelet/pods/334395e0-b09e-41b9-bb5c-4eb4b371eaa8/volumes"
Jul 01 19:43:43.752966 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:43.752953    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" path="/var/lib/kubelet/pods/492f1bb4-3f05-44d8-bb47-1a1b044cf510/volumes"
Jul 01 19:43:46.283810 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:46.283782    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:43:46.284194 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:46.284164    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 01 19:43:47.082009 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:47.081980    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:43:47.288154 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:47.288128    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:43:47.288625 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:47.288591    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 01 19:43:49.096098 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:49.096066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:43:56.285096 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:56.285055    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 01 19:43:57.289282 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:43:57.289240    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 01 19:44:06.284863 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:44:06.284823    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 01 19:44:07.288924 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:44:07.288879    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 01 19:44:16.284313 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:44:16.284268    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 01 19:44:17.288709 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:44:17.288669    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 01 19:44:26.285052 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:44:26.285022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:44:27.289478 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:44:27.289423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:46:20.438976 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:46:20.438860    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:46:20.443133 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:46:20.443114    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:51:20.469923 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:51:20.469822    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:51:20.475385 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:51:20.475363    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:52:24.042290 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042251    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"]
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042681    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kube-rbac-proxy"
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042694    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kube-rbac-proxy"
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042712    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kube-rbac-proxy"
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042718    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kube-rbac-proxy"
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042737    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container"
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042743    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container"
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042749    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container"
Jul 01 19:52:24.042781 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042754    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container"
Jul 01 19:52:24.043062 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042807    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kserve-container"
Jul 01 19:52:24.043062 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042817    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kube-rbac-proxy"
Jul 01 19:52:24.043062 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042823    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="492f1bb4-3f05-44d8-bb47-1a1b044cf510" containerName="kserve-container"
Jul 01 19:52:24.043062 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.042831    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="334395e0-b09e-41b9-bb5c-4eb4b371eaa8" containerName="kube-rbac-proxy"
Jul 01 19:52:24.051969 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.051932    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.053334 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.053305    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"]
Jul 01 19:52:24.053630 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.053551    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container" containerID="cri-o://61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa" gracePeriod=30
Jul 01 19:52:24.053740 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.053648    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kube-rbac-proxy" containerID="cri-o://1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a" gracePeriod=30
Jul 01 19:52:24.055041 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.055000    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8003b-predictor-serving-cert\""
Jul 01 19:52:24.055201 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.055040    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8003b-kube-rbac-proxy-sar-config\""
Jul 01 19:52:24.064184 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.064154    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"]
Jul 01 19:52:24.089728 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.089697    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 01 19:52:24.103703 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.103678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjwc\" (UniqueName: \"kubernetes.io/projected/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-api-access-sfjwc\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.103827 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.103749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.103827 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.103791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.137726 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.137701    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"]
Jul 01 19:52:24.141324 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.141309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.144811 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.144794    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8003b-predictor-serving-cert\""
Jul 01 19:52:24.145065 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.145044    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8003b-kube-rbac-proxy-sar-config\""
Jul 01 19:52:24.152811 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.152790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"]
Jul 01 19:52:24.196932 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.196907    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"]
Jul 01 19:52:24.197144 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.197103    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" containerID="cri-o://9f349e5ea0ee9132a48c3e4cc5d801b0ea432b1cdacc7b81ab91374f147b2d32" gracePeriod=30
Jul 01 19:52:24.197259 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.197140    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kube-rbac-proxy" containerID="cri-o://ae9c6a03f382b4ecd819d9fed8b7c9a0aeb84570a15b32e42f253d82d6ec602b" gracePeriod=30
Jul 01 19:52:24.204840 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.204819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1255d853-0fd5-4a41-9546-ddd0cabb2554-proxy-tls\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.204940 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.204863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.205008 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.204927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.205008 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.204986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72wd\" (UniqueName: \"kubernetes.io/projected/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-api-access-h72wd\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.205104 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.205085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjwc\" (UniqueName: \"kubernetes.io/projected/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-api-access-sfjwc\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.205157 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:24.205111    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8003b-predictor-serving-cert: secret "success-200-isvc-8003b-predictor-serving-cert" not found
Jul 01 19:52:24.205195 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.205144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.205195 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:24.205183    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls podName:a57e3829-7446-4fe1-865f-6b6745092bb7 nodeName:}" failed. No retries permitted until 2026-07-01 19:52:24.705162216 +0000 UTC m=+1871.464688930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls") pod "success-200-isvc-8003b-predictor-7d597dc584-hxm4g" (UID: "a57e3829-7446-4fe1-865f-6b6745092bb7") : secret "success-200-isvc-8003b-predictor-serving-cert" not found
Jul 01 19:52:24.205492 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.205473    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.221363 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.221339    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjwc\" (UniqueName: \"kubernetes.io/projected/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-api-access-sfjwc\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.306495 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.306416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.306601 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.306526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1255d853-0fd5-4a41-9546-ddd0cabb2554-proxy-tls\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.306651 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.306607    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72wd\" (UniqueName: \"kubernetes.io/projected/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-api-access-h72wd\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.307053 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.307029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.308991 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.308968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1255d853-0fd5-4a41-9546-ddd0cabb2554-proxy-tls\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.314875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.314852    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72wd\" (UniqueName: \"kubernetes.io/projected/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-api-access-h72wd\") pod \"error-404-isvc-8003b-predictor-69459f667c-ml85h\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") " pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.452022 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.451992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.585411 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.585389    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"]
Jul 01 19:52:24.587240 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:52:24.587208    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1255d853_0fd5_4a41_9546_ddd0cabb2554.slice/crio-8b4d3f07fc1a90256f7e4dedcb2ea41ea27c5011b79c10a97e09a5248f72bbc4 WatchSource:0}: Error finding container 8b4d3f07fc1a90256f7e4dedcb2ea41ea27c5011b79c10a97e09a5248f72bbc4: Status 404 returned error can't find the container with id 8b4d3f07fc1a90256f7e4dedcb2ea41ea27c5011b79c10a97e09a5248f72bbc4
Jul 01 19:52:24.709617 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.709594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.711682 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.711659    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls\") pod \"success-200-isvc-8003b-predictor-7d597dc584-hxm4g\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") " pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:24.768301 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.768270    2576 generic.go:356] "Generic (PLEG): container finished" podID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerID="ae9c6a03f382b4ecd819d9fed8b7c9a0aeb84570a15b32e42f253d82d6ec602b" exitCode=2
Jul 01 19:52:24.768418 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.768339    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" event={"ID":"f97ab1d3-1f52-4493-87ec-fcc966e68abc","Type":"ContainerDied","Data":"ae9c6a03f382b4ecd819d9fed8b7c9a0aeb84570a15b32e42f253d82d6ec602b"}
Jul 01 19:52:24.770146 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.770122    2576 generic.go:356] "Generic (PLEG): container finished" podID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerID="1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a" exitCode=2
Jul 01 19:52:24.770253 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.770198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" event={"ID":"b8128d6a-7d30-43d4-b37b-2cb6d3256c13","Type":"ContainerDied","Data":"1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a"}
Jul 01 19:52:24.772020 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.771997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" event={"ID":"1255d853-0fd5-4a41-9546-ddd0cabb2554","Type":"ContainerStarted","Data":"727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24"}
Jul 01 19:52:24.772125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.772042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" event={"ID":"1255d853-0fd5-4a41-9546-ddd0cabb2554","Type":"ContainerStarted","Data":"346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c"}
Jul 01 19:52:24.772125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.772068    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.772125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.772087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" event={"ID":"1255d853-0fd5-4a41-9546-ddd0cabb2554","Type":"ContainerStarted","Data":"8b4d3f07fc1a90256f7e4dedcb2ea41ea27c5011b79c10a97e09a5248f72bbc4"}
Jul 01 19:52:24.772125 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.772115    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:24.773709 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.773686    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 01 19:52:24.789945 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.789907    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podStartSLOduration=0.789893462 podStartE2EDuration="789.893462ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:52:24.788403503 +0000 UTC m=+1871.547930224" watchObservedRunningTime="2026-07-01 19:52:24.789893462 +0000 UTC m=+1871.549420183"
Jul 01 19:52:24.974050 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:24.973962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:25.108561 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:25.108538    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"]
Jul 01 19:52:25.110347 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:52:25.110321    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57e3829_7446_4fe1_865f_6b6745092bb7.slice/crio-efc156bd374df4d1b4205c35d7a8805edd6f129b213c16dcbd9bfd433b929d1e WatchSource:0}: Error finding container efc156bd374df4d1b4205c35d7a8805edd6f129b213c16dcbd9bfd433b929d1e: Status 404 returned error can't find the container with id efc156bd374df4d1b4205c35d7a8805edd6f129b213c16dcbd9bfd433b929d1e
Jul 01 19:52:25.778343 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:25.778306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" event={"ID":"a57e3829-7446-4fe1-865f-6b6745092bb7","Type":"ContainerStarted","Data":"1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9"}
Jul 01 19:52:25.778547 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:25.778365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" event={"ID":"a57e3829-7446-4fe1-865f-6b6745092bb7","Type":"ContainerStarted","Data":"91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d"}
Jul 01 19:52:25.778547 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:25.778394    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" event={"ID":"a57e3829-7446-4fe1-865f-6b6745092bb7","Type":"ContainerStarted","Data":"efc156bd374df4d1b4205c35d7a8805edd6f129b213c16dcbd9bfd433b929d1e"}
Jul 01 19:52:25.778674 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:25.778572    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 01 19:52:25.797329 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:25.797281    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podStartSLOduration=1.797266875 podStartE2EDuration="1.797266875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:52:25.795953704 +0000 UTC m=+1872.555480426" watchObservedRunningTime="2026-07-01 19:52:25.797266875 +0000 UTC m=+1872.556793596"
Jul 01 19:52:26.783258 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:26.783223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:26.783634 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:26.783284    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:26.784664 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:26.784634    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 01 19:52:27.076945 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.076858    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Jul 01 19:52:27.081192 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.081165    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 01 19:52:27.508008 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.507983    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:52:27.637387 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.637331    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8b8x\" (UniqueName: \"kubernetes.io/projected/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-api-access-t8b8x\") pod \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") "
Jul 01 19:52:27.637387 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.637365    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls\") pod \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") "
Jul 01 19:52:27.637617 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.637413    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-rbac-proxy-sar-config\") pod \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\" (UID: \"b8128d6a-7d30-43d4-b37b-2cb6d3256c13\") "
Jul 01 19:52:27.637813 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.637788    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8128d6a-7d30-43d4-b37b-2cb6d3256c13" (UID: "b8128d6a-7d30-43d4-b37b-2cb6d3256c13"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:52:27.639387 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.639367    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-api-access-t8b8x" (OuterVolumeSpecName: "kube-api-access-t8b8x") pod "b8128d6a-7d30-43d4-b37b-2cb6d3256c13" (UID: "b8128d6a-7d30-43d4-b37b-2cb6d3256c13"). InnerVolumeSpecName "kube-api-access-t8b8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:52:27.639476 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.639369    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8128d6a-7d30-43d4-b37b-2cb6d3256c13" (UID: "b8128d6a-7d30-43d4-b37b-2cb6d3256c13"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:52:27.738704 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.738679    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8b8x\" (UniqueName: \"kubernetes.io/projected/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-api-access-t8b8x\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:27.738704 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.738704    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:27.738838 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.738713    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8128d6a-7d30-43d4-b37b-2cb6d3256c13-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:27.789779 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.789748    2576 generic.go:356] "Generic (PLEG): container finished" podID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerID="9f349e5ea0ee9132a48c3e4cc5d801b0ea432b1cdacc7b81ab91374f147b2d32" exitCode=0
Jul 01 19:52:27.790096 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.789818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" event={"ID":"f97ab1d3-1f52-4493-87ec-fcc966e68abc","Type":"ContainerDied","Data":"9f349e5ea0ee9132a48c3e4cc5d801b0ea432b1cdacc7b81ab91374f147b2d32"}
Jul 01 19:52:27.791259 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.791240    2576 generic.go:356] "Generic (PLEG): container finished" podID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerID="61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa" exitCode=0
Jul 01 19:52:27.791320 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.791288    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"
Jul 01 19:52:27.791357 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.791315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" event={"ID":"b8128d6a-7d30-43d4-b37b-2cb6d3256c13","Type":"ContainerDied","Data":"61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa"}
Jul 01 19:52:27.791395 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.791360    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9" event={"ID":"b8128d6a-7d30-43d4-b37b-2cb6d3256c13","Type":"ContainerDied","Data":"47c100e1fe6eff12ac493ceb9cd83c009fc8b520d192ca95ce0397eab8391181"}
Jul 01 19:52:27.791395 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.791390    2576 scope.go:117] "RemoveContainer" containerID="1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a"
Jul 01 19:52:27.791875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.791847    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 01 19:52:27.800207 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.800193    2576 scope.go:117] "RemoveContainer" containerID="61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa"
Jul 01 19:52:27.808530 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.808510    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"]
Jul 01 19:52:27.808622 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.808592    2576 scope.go:117] "RemoveContainer" containerID="1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a"
Jul 01 19:52:27.808867 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:27.808846    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a\": container with ID starting with 1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a not found: ID does not exist" containerID="1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a"
Jul 01 19:52:27.808945 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.808872    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a"} err="failed to get container status \"1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a\": rpc error: code = NotFound desc = could not find container \"1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a\": container with ID starting with 1960d5e5ea776ac92124604f8a57f0c0df41babe8cc1c52b32d2668fc1719f4a not found: ID does not exist"
Jul 01 19:52:27.808945 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.808886    2576 scope.go:117] "RemoveContainer" containerID="61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa"
Jul 01 19:52:27.809103 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:27.809082    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa\": container with ID starting with 61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa not found: ID does not exist" containerID="61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa"
Jul 01 19:52:27.809200 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.809103    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa"} err="failed to get container status \"61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa\": rpc error: code = NotFound desc = could not find container \"61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa\": container with ID starting with 61533e70e293ce953caba4459e7b38cef14504fa58d774f1f71c2ae3336497fa not found: ID does not exist"
Jul 01 19:52:27.810499 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:27.810431    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-25c73-predictor-746989f766-6kkr9"]
Jul 01 19:52:28.040516 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.040489    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:52:28.141930 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.141897    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97ab1d3-1f52-4493-87ec-fcc966e68abc-proxy-tls\") pod \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") "
Jul 01 19:52:28.142053 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.141941    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-rbac-proxy-sar-config\") pod \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") "
Jul 01 19:52:28.142053 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.141999    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpz8g\" (UniqueName: \"kubernetes.io/projected/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-api-access-tpz8g\") pod \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\" (UID: \"f97ab1d3-1f52-4493-87ec-fcc966e68abc\") "
Jul 01 19:52:28.142398 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.142372    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f97ab1d3-1f52-4493-87ec-fcc966e68abc" (UID: "f97ab1d3-1f52-4493-87ec-fcc966e68abc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:52:28.143978 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.143949    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97ab1d3-1f52-4493-87ec-fcc966e68abc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f97ab1d3-1f52-4493-87ec-fcc966e68abc" (UID: "f97ab1d3-1f52-4493-87ec-fcc966e68abc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:52:28.144056 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.143982    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-api-access-tpz8g" (OuterVolumeSpecName: "kube-api-access-tpz8g") pod "f97ab1d3-1f52-4493-87ec-fcc966e68abc" (UID: "f97ab1d3-1f52-4493-87ec-fcc966e68abc"). InnerVolumeSpecName "kube-api-access-tpz8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:52:28.243254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.243190    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f97ab1d3-1f52-4493-87ec-fcc966e68abc-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:28.243254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.243217    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:28.243254 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.243230    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpz8g\" (UniqueName: \"kubernetes.io/projected/f97ab1d3-1f52-4493-87ec-fcc966e68abc-kube-api-access-tpz8g\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:28.798441 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.798403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf" event={"ID":"f97ab1d3-1f52-4493-87ec-fcc966e68abc","Type":"ContainerDied","Data":"bd5df3b25d4e8f757ee1e09e1163212768a35f1ebe6a122c094ef65f7b36566d"}
Jul 01 19:52:28.798919 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.798467    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"
Jul 01 19:52:28.798919 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.798487    2576 scope.go:117] "RemoveContainer" containerID="ae9c6a03f382b4ecd819d9fed8b7c9a0aeb84570a15b32e42f253d82d6ec602b"
Jul 01 19:52:28.807316 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.807294    2576 scope.go:117] "RemoveContainer" containerID="9f349e5ea0ee9132a48c3e4cc5d801b0ea432b1cdacc7b81ab91374f147b2d32"
Jul 01 19:52:28.820287 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.820260    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"]
Jul 01 19:52:28.823920 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:28.823889    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-25c73-predictor-6ddf4c5487-md7nf"]
Jul 01 19:52:29.751938 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:29.751903    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" path="/var/lib/kubelet/pods/b8128d6a-7d30-43d4-b37b-2cb6d3256c13/volumes"
Jul 01 19:52:29.752336 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:29.752323    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" path="/var/lib/kubelet/pods/f97ab1d3-1f52-4493-87ec-fcc966e68abc/volumes"
Jul 01 19:52:30.782724 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:30.782693    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:52:30.783102 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:30.783075    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 01 19:52:32.797033 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:32.797005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:52:32.797564 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:32.797534    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 01 19:52:40.784000 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:40.783916    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 01 19:52:42.797824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:42.797787    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 01 19:52:50.783496 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:50.783436    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 01 19:52:52.797693 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:52.797654    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 01 19:52:53.964640 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:53.964609    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"]
Jul 01 19:52:53.965026 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:53.964901    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" containerID="cri-o://018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907" gracePeriod=30
Jul 01 19:52:53.965026 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:53.964902    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kube-rbac-proxy" containerID="cri-o://b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944" gracePeriod=30
Jul 01 19:52:54.036076 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.036040    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"]
Jul 01 19:52:54.036339 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.036310    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" containerID="cri-o://7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b" gracePeriod=30
Jul 01 19:52:54.036429 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.036327    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kube-rbac-proxy" containerID="cri-o://fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea" gracePeriod=30
Jul 01 19:52:54.051382 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051358    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"]
Jul 01 19:52:54.051782 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051768    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container"
Jul 01 19:52:54.051847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051784    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container"
Jul 01 19:52:54.051847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051797    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container"
Jul 01 19:52:54.051847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051803    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container"
Jul 01 19:52:54.051847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051819    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kube-rbac-proxy"
Jul 01 19:52:54.051847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051825    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kube-rbac-proxy"
Jul 01 19:52:54.051847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051837    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kube-rbac-proxy"
Jul 01 19:52:54.051847 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051842    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kube-rbac-proxy"
Jul 01 19:52:54.052089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051900    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kserve-container"
Jul 01 19:52:54.052089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051907    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kserve-container"
Jul 01 19:52:54.052089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051915    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8128d6a-7d30-43d4-b37b-2cb6d3256c13" containerName="kube-rbac-proxy"
Jul 01 19:52:54.052089 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.051921    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97ab1d3-1f52-4493-87ec-fcc966e68abc" containerName="kube-rbac-proxy"
Jul 01 19:52:54.055288 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.055271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.058044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.057902    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-00068-predictor-serving-cert\""
Jul 01 19:52:54.058044 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.057997    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-00068-kube-rbac-proxy-sar-config\""
Jul 01 19:52:54.063842 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.063821    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"]
Jul 01 19:52:54.142968 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.141878    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"]
Jul 01 19:52:54.146161 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.146134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.148806 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.148778    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-00068-predictor-serving-cert\""
Jul 01 19:52:54.148944 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.148867    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-00068-kube-rbac-proxy-sar-config\""
Jul 01 19:52:54.154927 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.154906    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"]
Jul 01 19:52:54.160906 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.160873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k85g\" (UniqueName: \"kubernetes.io/projected/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-api-access-6k85g\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.161028 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.160942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.161092 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.161025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7fd533-c9ad-40ec-b822-fe5037faae02-proxy-tls\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.261824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.261754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7fd533-c9ad-40ec-b822-fe5037faae02-proxy-tls\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.261824 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.261816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k85g\" (UniqueName: \"kubernetes.io/projected/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-api-access-6k85g\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.262013 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.261867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.262013 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.261898    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.262013 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.261934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtbp8\" (UniqueName: \"kubernetes.io/projected/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-api-access-gtbp8\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.262171 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.262023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.262644 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.262625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.264154 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.264130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7fd533-c9ad-40ec-b822-fe5037faae02-proxy-tls\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.269825 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.269801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k85g\" (UniqueName: \"kubernetes.io/projected/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-api-access-6k85g\") pod \"success-200-isvc-00068-predictor-cd7ff766-4rlln\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") " pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.363098 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.363067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.363246 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.363103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.363246 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.363134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtbp8\" (UniqueName: \"kubernetes.io/projected/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-api-access-gtbp8\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.363246 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:54.363225    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-00068-predictor-serving-cert: secret "error-404-isvc-00068-predictor-serving-cert" not found
Jul 01 19:52:54.363396 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:54.363312    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls podName:036888f0-9d4c-4c8f-8d5b-8a9937c823b8 nodeName:}" failed. No retries permitted until 2026-07-01 19:52:54.863290108 +0000 UTC m=+1901.622816825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls") pod "error-404-isvc-00068-predictor-589d75fd87-jdckw" (UID: "036888f0-9d4c-4c8f-8d5b-8a9937c823b8") : secret "error-404-isvc-00068-predictor-serving-cert" not found
Jul 01 19:52:54.363795 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.363776    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.365999 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.365983    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.371802 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.371782    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtbp8\" (UniqueName: \"kubernetes.io/projected/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-api-access-gtbp8\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.504797 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.504771    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"]
Jul 01 19:52:54.506654 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:52:54.506628    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7fd533_c9ad_40ec_b822_fe5037faae02.slice/crio-20f133fc2eaaa8e6741bb1dce56c1a3207512c742f6829f053bacf46c512341d WatchSource:0}: Error finding container 20f133fc2eaaa8e6741bb1dce56c1a3207512c742f6829f053bacf46c512341d: Status 404 returned error can't find the container with id 20f133fc2eaaa8e6741bb1dce56c1a3207512c742f6829f053bacf46c512341d
Jul 01 19:52:54.869112 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.869081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.871349 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.871328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls\") pod \"error-404-isvc-00068-predictor-589d75fd87-jdckw\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") " pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:54.930186 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.930153    2576 generic.go:356] "Generic (PLEG): container finished" podID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerID="b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944" exitCode=2
Jul 01 19:52:54.930331 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.930225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" event={"ID":"f7e25b7d-d3e1-47cd-a289-1e3df644af1c","Type":"ContainerDied","Data":"b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944"}
Jul 01 19:52:54.931922 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.931903    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerID="fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea" exitCode=2
Jul 01 19:52:54.932021 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.931979    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" event={"ID":"3a79a16d-57d1-4d3c-ae44-fa833c56ef89","Type":"ContainerDied","Data":"fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea"}
Jul 01 19:52:54.933558 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.933537    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" event={"ID":"1d7fd533-c9ad-40ec-b822-fe5037faae02","Type":"ContainerStarted","Data":"d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d"}
Jul 01 19:52:54.933660 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.933589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" event={"ID":"1d7fd533-c9ad-40ec-b822-fe5037faae02","Type":"ContainerStarted","Data":"48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f"}
Jul 01 19:52:54.933660 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.933621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" event={"ID":"1d7fd533-c9ad-40ec-b822-fe5037faae02","Type":"ContainerStarted","Data":"20f133fc2eaaa8e6741bb1dce56c1a3207512c742f6829f053bacf46c512341d"}
Jul 01 19:52:54.933771 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.933696    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:54.951510 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:54.951459    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podStartSLOduration=0.9514271 podStartE2EDuration="951.4271ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:52:54.950784493 +0000 UTC m=+1901.710311239" watchObservedRunningTime="2026-07-01 19:52:54.9514271 +0000 UTC m=+1901.710953821"
Jul 01 19:52:55.058068 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.058041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:55.200335 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.200310    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"]
Jul 01 19:52:55.203073 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:52:55.203051    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036888f0_9d4c_4c8f_8d5b_8a9937c823b8.slice/crio-a2c14eb374b82288bf5872ec8458b29727f4fd70b90c6113ee11437966935cb7 WatchSource:0}: Error finding container a2c14eb374b82288bf5872ec8458b29727f4fd70b90c6113ee11437966935cb7: Status 404 returned error can't find the container with id a2c14eb374b82288bf5872ec8458b29727f4fd70b90c6113ee11437966935cb7
Jul 01 19:52:55.942748 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.942707    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" event={"ID":"036888f0-9d4c-4c8f-8d5b-8a9937c823b8","Type":"ContainerStarted","Data":"3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec"}
Jul 01 19:52:55.942948 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.942782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" event={"ID":"036888f0-9d4c-4c8f-8d5b-8a9937c823b8","Type":"ContainerStarted","Data":"ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523"}
Jul 01 19:52:55.942948 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.942818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" event={"ID":"036888f0-9d4c-4c8f-8d5b-8a9937c823b8","Type":"ContainerStarted","Data":"a2c14eb374b82288bf5872ec8458b29727f4fd70b90c6113ee11437966935cb7"}
Jul 01 19:52:55.946109 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.944909    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:52:55.946109 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.944966    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:55.946109 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.945000    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:52:55.946109 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.945072    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 01 19:52:55.946871 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.946837    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 01 19:52:55.962425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:55.962372    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podStartSLOduration=1.9623596129999998 podStartE2EDuration="1.962359613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:52:55.959354809 +0000 UTC m=+1902.718881542" watchObservedRunningTime="2026-07-01 19:52:55.962359613 +0000 UTC m=+1902.721886334"
Jul 01 19:52:56.280091 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:56.280007    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 01 19:52:56.284427 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:56.284394    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 01 19:52:56.954678 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:56.954626    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 01 19:52:56.955342 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:56.954712    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 01 19:52:57.284404 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.284367    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 01 19:52:57.289074 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.289047    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 01 19:52:57.538340 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.538269    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:52:57.701158 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.701116    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxlbh\" (UniqueName: \"kubernetes.io/projected/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-api-access-bxlbh\") pod \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") "
Jul 01 19:52:57.701347 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.701209    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-rbac-proxy-sar-config\") pod \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") "
Jul 01 19:52:57.701347 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.701301    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-proxy-tls\") pod \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\" (UID: \"f7e25b7d-d3e1-47cd-a289-1e3df644af1c\") "
Jul 01 19:52:57.701648 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.701611    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7e25b7d-d3e1-47cd-a289-1e3df644af1c" (UID: "f7e25b7d-d3e1-47cd-a289-1e3df644af1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:52:57.703162 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.703129    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-api-access-bxlbh" (OuterVolumeSpecName: "kube-api-access-bxlbh") pod "f7e25b7d-d3e1-47cd-a289-1e3df644af1c" (UID: "f7e25b7d-d3e1-47cd-a289-1e3df644af1c"). InnerVolumeSpecName "kube-api-access-bxlbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:52:57.703551 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.703527    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7e25b7d-d3e1-47cd-a289-1e3df644af1c" (UID: "f7e25b7d-d3e1-47cd-a289-1e3df644af1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:52:57.778316 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.778265    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:52:57.802659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.802591    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:57.802659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.802625    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxlbh\" (UniqueName: \"kubernetes.io/projected/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-api-access-bxlbh\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:57.802659 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.802639    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7e25b7d-d3e1-47cd-a289-1e3df644af1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:57.903354 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.903323    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-proxy-tls\") pod \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") "
Jul 01 19:52:57.903539 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.903369    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-rbac-proxy-sar-config\") pod \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") "
Jul 01 19:52:57.903539 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.903431    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nf7v\" (UniqueName: \"kubernetes.io/projected/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-api-access-4nf7v\") pod \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\" (UID: \"3a79a16d-57d1-4d3c-ae44-fa833c56ef89\") "
Jul 01 19:52:57.903823 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.903792    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a79a16d-57d1-4d3c-ae44-fa833c56ef89" (UID: "3a79a16d-57d1-4d3c-ae44-fa833c56ef89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:52:57.905528 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.905502    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a79a16d-57d1-4d3c-ae44-fa833c56ef89" (UID: "3a79a16d-57d1-4d3c-ae44-fa833c56ef89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:52:57.905611 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.905563    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-api-access-4nf7v" (OuterVolumeSpecName: "kube-api-access-4nf7v") pod "3a79a16d-57d1-4d3c-ae44-fa833c56ef89" (UID: "3a79a16d-57d1-4d3c-ae44-fa833c56ef89"). InnerVolumeSpecName "kube-api-access-4nf7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:52:57.959296 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.959265    2576 generic.go:356] "Generic (PLEG): container finished" podID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerID="018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907" exitCode=0
Jul 01 19:52:57.959445 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.959323    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"
Jul 01 19:52:57.959445 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.959355    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" event={"ID":"f7e25b7d-d3e1-47cd-a289-1e3df644af1c","Type":"ContainerDied","Data":"018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907"}
Jul 01 19:52:57.959445 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.959421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp" event={"ID":"f7e25b7d-d3e1-47cd-a289-1e3df644af1c","Type":"ContainerDied","Data":"39f91762f0fc41b457950321457b85bbd818275f3640d52c3a8f84ce717c3b2c"}
Jul 01 19:52:57.959606 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.959468    2576 scope.go:117] "RemoveContainer" containerID="b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944"
Jul 01 19:52:57.963269 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.963236    2576 generic.go:356] "Generic (PLEG): container finished" podID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerID="7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b" exitCode=0
Jul 01 19:52:57.963429 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.963411    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" event={"ID":"3a79a16d-57d1-4d3c-ae44-fa833c56ef89","Type":"ContainerDied","Data":"7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b"}
Jul 01 19:52:57.963554 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.963506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf" event={"ID":"3a79a16d-57d1-4d3c-ae44-fa833c56ef89","Type":"ContainerDied","Data":"645637b51db165c4adcea350f9811a5ea17a496d9ae5353f75977131d1456516"}
Jul 01 19:52:57.963797 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.963775    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 01 19:52:57.964364 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.964000    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"
Jul 01 19:52:57.970391 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.970374    2576 scope.go:117] "RemoveContainer" containerID="018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907"
Jul 01 19:52:57.978711 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.978652    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"]
Jul 01 19:52:57.979786 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.979766    2576 scope.go:117] "RemoveContainer" containerID="b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944"
Jul 01 19:52:57.980020 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:57.980003    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944\": container with ID starting with b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944 not found: ID does not exist" containerID="b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944"
Jul 01 19:52:57.980085 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.980026    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944"} err="failed to get container status \"b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944\": rpc error: code = NotFound desc = could not find container \"b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944\": container with ID starting with b9a970a5578d9526565f033ae2738c769f54711b3373ca0a693ccf2e11654944 not found: ID does not exist"
Jul 01 19:52:57.980085 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.980040    2576 scope.go:117] "RemoveContainer" containerID="018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907"
Jul 01 19:52:57.980266 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:57.980246    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907\": container with ID starting with 018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907 not found: ID does not exist" containerID="018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907"
Jul 01 19:52:57.980309 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.980274    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907"} err="failed to get container status \"018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907\": rpc error: code = NotFound desc = could not find container \"018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907\": container with ID starting with 018ed1c498c698f1237e997ef0b549160a37f03ce9ec6202cbbfda814277f907 not found: ID does not exist"
Jul 01 19:52:57.980309 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.980290    2576 scope.go:117] "RemoveContainer" containerID="fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea"
Jul 01 19:52:57.981865 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.981849    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f705b-predictor-d6ff57dff-d2dhp"]
Jul 01 19:52:57.987678 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.987663    2576 scope.go:117] "RemoveContainer" containerID="7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b"
Jul 01 19:52:57.991123 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.991104    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"]
Jul 01 19:52:57.995472 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.995439    2576 scope.go:117] "RemoveContainer" containerID="fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea"
Jul 01 19:52:57.995750 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:57.995732    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea\": container with ID starting with fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea not found: ID does not exist" containerID="fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea"
Jul 01 19:52:57.995805 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.995757    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea"} err="failed to get container status \"fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea\": rpc error: code = NotFound desc = could not find container \"fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea\": container with ID starting with fcf8db1f9f1f0a3ae2934b1db4df9f03de7bae5a84534ed506ccd46138a2d1ea not found: ID does not exist"
Jul 01 19:52:57.995805 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.995774    2576 scope.go:117] "RemoveContainer" containerID="7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b"
Jul 01 19:52:57.996005 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:52:57.995988    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b\": container with ID starting with 7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b not found: ID does not exist" containerID="7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b"
Jul 01 19:52:57.996048 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.996009    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b"} err="failed to get container status \"7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b\": rpc error: code = NotFound desc = could not find container \"7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b\": container with ID starting with 7286f52796d5ae6a8f1934a3029271e9a3b45397555c066251940082dc128c8b not found: ID does not exist"
Jul 01 19:52:57.996680 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:57.996664    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f705b-predictor-7759fdc7c4-5jkhf"]
Jul 01 19:52:58.004496 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:58.004477    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:58.004553 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:58.004497    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:58.004553 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:58.004508    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nf7v\" (UniqueName: \"kubernetes.io/projected/3a79a16d-57d1-4d3c-ae44-fa833c56ef89-kube-api-access-4nf7v\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:52:59.752874 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:59.752844    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" path="/var/lib/kubelet/pods/3a79a16d-57d1-4d3c-ae44-fa833c56ef89/volumes"
Jul 01 19:52:59.753262 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:52:59.753248    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" path="/var/lib/kubelet/pods/f7e25b7d-d3e1-47cd-a289-1e3df644af1c/volumes"
Jul 01 19:53:00.783766 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:00.783728    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 01 19:53:01.958653 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:01.958624    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:53:01.959138 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:01.959108    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 01 19:53:02.797458 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:02.797418    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 01 19:53:02.967719 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:02.967690    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:53:02.968191 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:02.968160    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 01 19:53:10.784426 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:10.784396    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:53:11.960029 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:11.959991    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 01 19:53:12.798489 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:12.798442    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:53:12.969179 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:12.969137    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 01 19:53:21.959672 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:21.959632    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 01 19:53:22.969017 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:22.968978    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 01 19:53:31.959251 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:31.959200    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 01 19:53:32.968888 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:32.968850    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 01 19:53:34.268210 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.268176    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"]
Jul 01 19:53:34.268731 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.268488    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container" containerID="cri-o://91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d" gracePeriod=30
Jul 01 19:53:34.268731 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.268539    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kube-rbac-proxy" containerID="cri-o://1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9" gracePeriod=30
Jul 01 19:53:34.282115 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282087    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"]
Jul 01 19:53:34.282730 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282711    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container"
Jul 01 19:53:34.282730 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282730    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container"
Jul 01 19:53:34.282898 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282747    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container"
Jul 01 19:53:34.282898 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282755    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container"
Jul 01 19:53:34.282898 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282779    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kube-rbac-proxy"
Jul 01 19:53:34.282898 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282787    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kube-rbac-proxy"
Jul 01 19:53:34.282898 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282803    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kube-rbac-proxy"
Jul 01 19:53:34.282898 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282811    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kube-rbac-proxy"
Jul 01 19:53:34.283205 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282916    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kserve-container"
Jul 01 19:53:34.283205 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282933    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e25b7d-d3e1-47cd-a289-1e3df644af1c" containerName="kube-rbac-proxy"
Jul 01 19:53:34.283205 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282943    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kserve-container"
Jul 01 19:53:34.283205 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.282954    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a79a16d-57d1-4d3c-ae44-fa833c56ef89" containerName="kube-rbac-proxy"
Jul 01 19:53:34.286713 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.286697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.289264 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.289239    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5e51f-predictor-serving-cert\""
Jul 01 19:53:34.289504 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.289486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5e51f-kube-rbac-proxy-sar-config\""
Jul 01 19:53:34.297530 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.297511    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"]
Jul 01 19:53:34.385209 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.385181    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"]
Jul 01 19:53:34.388767 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.388750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.392368 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.392348    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5e51f-kube-rbac-proxy-sar-config\""
Jul 01 19:53:34.392535 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.392517    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5e51f-predictor-serving-cert\""
Jul 01 19:53:34.404594 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.404572    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"]
Jul 01 19:53:34.409507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.409483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.409616 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.409561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3031d17b-7f58-4c88-b495-1e08101ac08f-proxy-tls\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.409699 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.409646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfjr\" (UniqueName: \"kubernetes.io/projected/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-api-access-xsfjr\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.448175 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.448143    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"]
Jul 01 19:53:34.449103 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.449066    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container" containerID="cri-o://346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c" gracePeriod=30
Jul 01 19:53:34.449395 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.449108    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kube-rbac-proxy" containerID="cri-o://727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24" gracePeriod=30
Jul 01 19:53:34.510575 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.510549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfjr\" (UniqueName: \"kubernetes.io/projected/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-api-access-xsfjr\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.510666 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.510588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.510666 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.510644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3031d17b-7f58-4c88-b495-1e08101ac08f-proxy-tls\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.510666 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.510663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc80bb1-12af-4722-b761-c7cf48940444-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.510862 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.510689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cv4\" (UniqueName: \"kubernetes.io/projected/4cc80bb1-12af-4722-b761-c7cf48940444-kube-api-access-c5cv4\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.510862 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.510793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.511257 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.511233    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.512878 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.512854    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3031d17b-7f58-4c88-b495-1e08101ac08f-proxy-tls\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.518439 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.518397    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfjr\" (UniqueName: \"kubernetes.io/projected/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-api-access-xsfjr\") pod \"success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") " pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.597507 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.597484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:34.611510 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.611486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc80bb1-12af-4722-b761-c7cf48940444-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.611623 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.611519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cv4\" (UniqueName: \"kubernetes.io/projected/4cc80bb1-12af-4722-b761-c7cf48940444-kube-api-access-c5cv4\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.611623 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.611572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.611750 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:53:34.611711    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-serving-cert: secret "error-404-isvc-5e51f-predictor-serving-cert" not found
Jul 01 19:53:34.611804 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:53:34.611760    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls podName:4cc80bb1-12af-4722-b761-c7cf48940444 nodeName:}" failed. No retries permitted until 2026-07-01 19:53:35.111746443 +0000 UTC m=+1941.871273142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls") pod "error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" (UID: "4cc80bb1-12af-4722-b761-c7cf48940444") : secret "error-404-isvc-5e51f-predictor-serving-cert" not found
Jul 01 19:53:34.612162 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.612141    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc80bb1-12af-4722-b761-c7cf48940444-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.623031 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.623002    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cv4\" (UniqueName: \"kubernetes.io/projected/4cc80bb1-12af-4722-b761-c7cf48940444-kube-api-access-c5cv4\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:34.734837 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:34.734805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"]
Jul 01 19:53:34.737824 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:53:34.737797    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3031d17b_7f58_4c88_b495_1e08101ac08f.slice/crio-00f40ac4af776ca43111212aa21dfcf4959285ee96898c067c12b16a845b4321 WatchSource:0}: Error finding container 00f40ac4af776ca43111212aa21dfcf4959285ee96898c067c12b16a845b4321: Status 404 returned error can't find the container with id 00f40ac4af776ca43111212aa21dfcf4959285ee96898c067c12b16a845b4321
Jul 01 19:53:35.116494 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.116441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:35.118786 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.118759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls\") pod \"error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") " pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:35.156117 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.156080    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" event={"ID":"3031d17b-7f58-4c88-b495-1e08101ac08f","Type":"ContainerStarted","Data":"1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a"}
Jul 01 19:53:35.156275 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.156151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" event={"ID":"3031d17b-7f58-4c88-b495-1e08101ac08f","Type":"ContainerStarted","Data":"361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa"}
Jul 01 19:53:35.156275 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.156198    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:35.156275 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.156226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" event={"ID":"3031d17b-7f58-4c88-b495-1e08101ac08f","Type":"ContainerStarted","Data":"00f40ac4af776ca43111212aa21dfcf4959285ee96898c067c12b16a845b4321"}
Jul 01 19:53:35.156275 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.156256    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:35.157999 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.157977    2576 generic.go:356] "Generic (PLEG): container finished" podID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerID="727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24" exitCode=2
Jul 01 19:53:35.158114 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.158058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" event={"ID":"1255d853-0fd5-4a41-9546-ddd0cabb2554","Type":"ContainerDied","Data":"727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24"}
Jul 01 19:53:35.158156 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.158107    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 01 19:53:35.159793 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.159766    2576 generic.go:356] "Generic (PLEG): container finished" podID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerID="1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9" exitCode=2
Jul 01 19:53:35.159877 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.159807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" event={"ID":"a57e3829-7446-4fe1-865f-6b6745092bb7","Type":"ContainerDied","Data":"1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9"}
Jul 01 19:53:35.299500 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.299469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:35.437425 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.437364    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podStartSLOduration=1.437340013 podStartE2EDuration="1.437340013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:53:35.199806915 +0000 UTC m=+1941.959333644" watchObservedRunningTime="2026-07-01 19:53:35.437340013 +0000 UTC m=+1942.196866735"
Jul 01 19:53:35.439421 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.439403    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"]
Jul 01 19:53:35.441700 ip-10-0-130-234 kubenswrapper[2576]: W0701 19:53:35.441652    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc80bb1_12af_4722_b761_c7cf48940444.slice/crio-07d50756d708d97e27cc571979a8db96d69e22c2bc93ed979822f20f1b83bfe7 WatchSource:0}: Error finding container 07d50756d708d97e27cc571979a8db96d69e22c2bc93ed979822f20f1b83bfe7: Status 404 returned error can't find the container with id 07d50756d708d97e27cc571979a8db96d69e22c2bc93ed979822f20f1b83bfe7
Jul 01 19:53:35.779821 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:35.779732    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 01 19:53:36.166801 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.166761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" event={"ID":"4cc80bb1-12af-4722-b761-c7cf48940444","Type":"ContainerStarted","Data":"2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d"}
Jul 01 19:53:36.166979 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.166831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" event={"ID":"4cc80bb1-12af-4722-b761-c7cf48940444","Type":"ContainerStarted","Data":"1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790"}
Jul 01 19:53:36.167046 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.167014    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 01 19:53:36.167179 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.167163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:36.167213 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.167205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" event={"ID":"4cc80bb1-12af-4722-b761-c7cf48940444","Type":"ContainerStarted","Data":"07d50756d708d97e27cc571979a8db96d69e22c2bc93ed979822f20f1b83bfe7"}
Jul 01 19:53:36.167249 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.167229    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:36.168392 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.168369    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 01 19:53:36.186175 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:36.186131    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podStartSLOduration=2.186118716 podStartE2EDuration="2.186118716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 19:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 19:53:36.18510543 +0000 UTC m=+1942.944632161" watchObservedRunningTime="2026-07-01 19:53:36.186118716 +0000 UTC m=+1942.945645437"
Jul 01 19:53:37.172433 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.172382    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 01 19:53:37.421778 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.421755    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:53:37.535857 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.535833    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfjwc\" (UniqueName: \"kubernetes.io/projected/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-api-access-sfjwc\") pod \"a57e3829-7446-4fe1-865f-6b6745092bb7\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") "
Jul 01 19:53:37.536003 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.535889    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-rbac-proxy-sar-config\") pod \"a57e3829-7446-4fe1-865f-6b6745092bb7\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") "
Jul 01 19:53:37.536072 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.536013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls\") pod \"a57e3829-7446-4fe1-865f-6b6745092bb7\" (UID: \"a57e3829-7446-4fe1-865f-6b6745092bb7\") "
Jul 01 19:53:37.536175 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.536158    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a57e3829-7446-4fe1-865f-6b6745092bb7" (UID: "a57e3829-7446-4fe1-865f-6b6745092bb7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:53:37.536327 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.536311    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:53:37.537859 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.537834    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a57e3829-7446-4fe1-865f-6b6745092bb7" (UID: "a57e3829-7446-4fe1-865f-6b6745092bb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:53:37.537987 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.537955    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-api-access-sfjwc" (OuterVolumeSpecName: "kube-api-access-sfjwc") pod "a57e3829-7446-4fe1-865f-6b6745092bb7" (UID: "a57e3829-7446-4fe1-865f-6b6745092bb7"). InnerVolumeSpecName "kube-api-access-sfjwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:53:37.637199 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.637159    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjwc\" (UniqueName: \"kubernetes.io/projected/a57e3829-7446-4fe1-865f-6b6745092bb7-kube-api-access-sfjwc\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:53:37.637199 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.637196    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a57e3829-7446-4fe1-865f-6b6745092bb7-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:53:37.991192 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:37.991167    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:53:38.142370 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.142221    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1255d853-0fd5-4a41-9546-ddd0cabb2554-proxy-tls\") pod \"1255d853-0fd5-4a41-9546-ddd0cabb2554\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") "
Jul 01 19:53:38.142370 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.142340    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-rbac-proxy-sar-config\") pod \"1255d853-0fd5-4a41-9546-ddd0cabb2554\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") "
Jul 01 19:53:38.142697 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.142381    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h72wd\" (UniqueName: \"kubernetes.io/projected/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-api-access-h72wd\") pod \"1255d853-0fd5-4a41-9546-ddd0cabb2554\" (UID: \"1255d853-0fd5-4a41-9546-ddd0cabb2554\") "
Jul 01 19:53:38.142766 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.142729    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1255d853-0fd5-4a41-9546-ddd0cabb2554" (UID: "1255d853-0fd5-4a41-9546-ddd0cabb2554"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 19:53:38.144593 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.144562    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1255d853-0fd5-4a41-9546-ddd0cabb2554-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1255d853-0fd5-4a41-9546-ddd0cabb2554" (UID: "1255d853-0fd5-4a41-9546-ddd0cabb2554"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 19:53:38.144708 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.144656    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-api-access-h72wd" (OuterVolumeSpecName: "kube-api-access-h72wd") pod "1255d853-0fd5-4a41-9546-ddd0cabb2554" (UID: "1255d853-0fd5-4a41-9546-ddd0cabb2554"). InnerVolumeSpecName "kube-api-access-h72wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 19:53:38.178792 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.178765    2576 generic.go:356] "Generic (PLEG): container finished" podID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerID="346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c" exitCode=0
Jul 01 19:53:38.179139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.178819    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"
Jul 01 19:53:38.179139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.178853    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" event={"ID":"1255d853-0fd5-4a41-9546-ddd0cabb2554","Type":"ContainerDied","Data":"346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c"}
Jul 01 19:53:38.179139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.178914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h" event={"ID":"1255d853-0fd5-4a41-9546-ddd0cabb2554","Type":"ContainerDied","Data":"8b4d3f07fc1a90256f7e4dedcb2ea41ea27c5011b79c10a97e09a5248f72bbc4"}
Jul 01 19:53:38.179139 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.178942    2576 scope.go:117] "RemoveContainer" containerID="727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24"
Jul 01 19:53:38.180567 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.180544    2576 generic.go:356] "Generic (PLEG): container finished" podID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerID="91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d" exitCode=0
Jul 01 19:53:38.180669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.180592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"
Jul 01 19:53:38.180669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.180600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" event={"ID":"a57e3829-7446-4fe1-865f-6b6745092bb7","Type":"ContainerDied","Data":"91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d"}
Jul 01 19:53:38.180669 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.180649    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g" event={"ID":"a57e3829-7446-4fe1-865f-6b6745092bb7","Type":"ContainerDied","Data":"efc156bd374df4d1b4205c35d7a8805edd6f129b213c16dcbd9bfd433b929d1e"}
Jul 01 19:53:38.188110 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.188091    2576 scope.go:117] "RemoveContainer" containerID="346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c"
Jul 01 19:53:38.196041 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.196025    2576 scope.go:117] "RemoveContainer" containerID="727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24"
Jul 01 19:53:38.196286 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:53:38.196268    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24\": container with ID starting with 727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24 not found: ID does not exist" containerID="727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24"
Jul 01 19:53:38.196339 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.196293    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24"} err="failed to get container status \"727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24\": rpc error: code = NotFound desc = could not find container \"727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24\": container with ID starting with 727004b125b8cdd73966b5e2809f1600c887bf11091e52e44246647909e99b24 not found: ID does not exist"
Jul 01 19:53:38.196339 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.196310    2576 scope.go:117] "RemoveContainer" containerID="346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c"
Jul 01 19:53:38.196528 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:53:38.196514    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c\": container with ID starting with 346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c not found: ID does not exist" containerID="346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c"
Jul 01 19:53:38.196567 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.196530    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c"} err="failed to get container status \"346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c\": rpc error: code = NotFound desc = could not find container \"346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c\": container with ID starting with 346beee09b4c3a168a001be88ad276881cbbec8bd0814c8f955927680523984c not found: ID does not exist"
Jul 01 19:53:38.196567 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.196541    2576 scope.go:117] "RemoveContainer" containerID="1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9"
Jul 01 19:53:38.196567 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.196551    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"]
Jul 01 19:53:38.199988 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.199969    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8003b-predictor-7d597dc584-hxm4g"]
Jul 01 19:53:38.204219 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.204197    2576 scope.go:117] "RemoveContainer" containerID="91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d"
Jul 01 19:53:38.210651 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.210613    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"]
Jul 01 19:53:38.212436 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.212414    2576 scope.go:117] "RemoveContainer" containerID="1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9"
Jul 01 19:53:38.212682 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:53:38.212665    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9\": container with ID starting with 1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9 not found: ID does not exist" containerID="1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9"
Jul 01 19:53:38.212743 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.212688    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9"} err="failed to get container status \"1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9\": rpc error: code = NotFound desc = could not find container \"1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9\": container with ID starting with 1540462ca956f422055711268e6aa034e824b0e5c5ad27f890f0656159e3cae9 not found: ID does not exist"
Jul 01 19:53:38.212743 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.212704    2576 scope.go:117] "RemoveContainer" containerID="91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d"
Jul 01 19:53:38.212964 ip-10-0-130-234 kubenswrapper[2576]: E0701 19:53:38.212940    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d\": container with ID starting with 91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d not found: ID does not exist" containerID="91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d"
Jul 01 19:53:38.213011 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.212975    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8003b-predictor-69459f667c-ml85h"]
Jul 01 19:53:38.213011 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.212971    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d"} err="failed to get container status \"91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d\": rpc error: code = NotFound desc = could not find container \"91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d\": container with ID starting with 91b0dba9de8cec955bb19202eebbb1a59083702ba565cba48fe5581c0682e38d not found: ID does not exist"
Jul 01 19:53:38.243525 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.243506    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:53:38.243525 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.243525    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h72wd\" (UniqueName: \"kubernetes.io/projected/1255d853-0fd5-4a41-9546-ddd0cabb2554-kube-api-access-h72wd\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:53:38.243625 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:38.243536    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1255d853-0fd5-4a41-9546-ddd0cabb2554-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 19:53:39.751804 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:39.751774    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" path="/var/lib/kubelet/pods/1255d853-0fd5-4a41-9546-ddd0cabb2554/volumes"
Jul 01 19:53:39.752236 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:39.752206    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" path="/var/lib/kubelet/pods/a57e3829-7446-4fe1-865f-6b6745092bb7/volumes"
Jul 01 19:53:41.171262 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:41.171230    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:53:41.171702 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:41.171680    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 01 19:53:41.960613 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:41.960584    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 19:53:42.176668 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:42.176639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:53:42.177142 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:42.177115    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 01 19:53:42.969348 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:42.969319    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 19:53:51.172399 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:51.172358    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 01 19:53:52.177802 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:53:52.177761    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 01 19:54:01.171875 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:54:01.171825    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 01 19:54:02.177939 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:54:02.177897    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 01 19:54:11.172157 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:54:11.172082    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 01 19:54:12.177390 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:54:12.177352    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 01 19:54:21.172623 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:54:21.172590    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 19:54:22.177598 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:54:22.177567    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 19:56:20.501047 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:56:20.500939    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 19:56:20.506622 ip-10-0-130-234 kubenswrapper[2576]: I0701 19:56:20.506596    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 20:01:20.530464 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:01:20.530333    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 20:01:20.541250 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:01:20.541230    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 20:02:49.211258 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.211221    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"]
Jul 01 20:02:49.211804 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.211519    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" containerID="cri-o://361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa" gracePeriod=30
Jul 01 20:02:49.211804 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.211584    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kube-rbac-proxy" containerID="cri-o://1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a" gracePeriod=30
Jul 01 20:02:49.270953 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.270923    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"]
Jul 01 20:02:49.271162 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.271140    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" containerID="cri-o://1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790" gracePeriod=30
Jul 01 20:02:49.271256 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.271167    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kube-rbac-proxy" containerID="cri-o://2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d" gracePeriod=30
Jul 01 20:02:49.807700 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.807666    2576 generic.go:356] "Generic (PLEG): container finished" podID="4cc80bb1-12af-4722-b761-c7cf48940444" containerID="2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d" exitCode=2
Jul 01 20:02:49.807888 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.807732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" event={"ID":"4cc80bb1-12af-4722-b761-c7cf48940444","Type":"ContainerDied","Data":"2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d"}
Jul 01 20:02:49.809538 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.809514    2576 generic.go:356] "Generic (PLEG): container finished" podID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerID="1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a" exitCode=2
Jul 01 20:02:49.809617 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:49.809573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" event={"ID":"3031d17b-7f58-4c88-b495-1e08101ac08f","Type":"ContainerDied","Data":"1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a"}
Jul 01 20:02:51.168048 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:51.168010    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Jul 01 20:02:51.171658 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:51.171634    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 01 20:02:52.172766 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.172713    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Jul 01 20:02:52.177305 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.177274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 01 20:02:52.292012 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.291958    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 20:02:52.294833 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.294808    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 20:02:52.392202 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392172    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls\") pod \"4cc80bb1-12af-4722-b761-c7cf48940444\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") "
Jul 01 20:02:52.392340 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392240    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsfjr\" (UniqueName: \"kubernetes.io/projected/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-api-access-xsfjr\") pod \"3031d17b-7f58-4c88-b495-1e08101ac08f\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") "
Jul 01 20:02:52.392340 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392294    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc80bb1-12af-4722-b761-c7cf48940444-kube-rbac-proxy-sar-config\") pod \"4cc80bb1-12af-4722-b761-c7cf48940444\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") "
Jul 01 20:02:52.392485 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-rbac-proxy-sar-config\") pod \"3031d17b-7f58-4c88-b495-1e08101ac08f\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") "
Jul 01 20:02:52.392485 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392390    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3031d17b-7f58-4c88-b495-1e08101ac08f-proxy-tls\") pod \"3031d17b-7f58-4c88-b495-1e08101ac08f\" (UID: \"3031d17b-7f58-4c88-b495-1e08101ac08f\") "
Jul 01 20:02:52.392485 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392414    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5cv4\" (UniqueName: \"kubernetes.io/projected/4cc80bb1-12af-4722-b761-c7cf48940444-kube-api-access-c5cv4\") pod \"4cc80bb1-12af-4722-b761-c7cf48940444\" (UID: \"4cc80bb1-12af-4722-b761-c7cf48940444\") "
Jul 01 20:02:52.392787 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392755    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3031d17b-7f58-4c88-b495-1e08101ac08f" (UID: "3031d17b-7f58-4c88-b495-1e08101ac08f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:02:52.392887 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.392760    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc80bb1-12af-4722-b761-c7cf48940444-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4cc80bb1-12af-4722-b761-c7cf48940444" (UID: "4cc80bb1-12af-4722-b761-c7cf48940444"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:02:52.394525 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.394503    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3031d17b-7f58-4c88-b495-1e08101ac08f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3031d17b-7f58-4c88-b495-1e08101ac08f" (UID: "3031d17b-7f58-4c88-b495-1e08101ac08f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:02:52.394648 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.394531    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc80bb1-12af-4722-b761-c7cf48940444-kube-api-access-c5cv4" (OuterVolumeSpecName: "kube-api-access-c5cv4") pod "4cc80bb1-12af-4722-b761-c7cf48940444" (UID: "4cc80bb1-12af-4722-b761-c7cf48940444"). InnerVolumeSpecName "kube-api-access-c5cv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:02:52.394765 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.394749    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-api-access-xsfjr" (OuterVolumeSpecName: "kube-api-access-xsfjr") pod "3031d17b-7f58-4c88-b495-1e08101ac08f" (UID: "3031d17b-7f58-4c88-b495-1e08101ac08f"). InnerVolumeSpecName "kube-api-access-xsfjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:02:52.394808 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.394753    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4cc80bb1-12af-4722-b761-c7cf48940444" (UID: "4cc80bb1-12af-4722-b761-c7cf48940444"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:02:52.494001 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.493962    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cc80bb1-12af-4722-b761-c7cf48940444-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:02:52.494001 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.494000    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsfjr\" (UniqueName: \"kubernetes.io/projected/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-api-access-xsfjr\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:02:52.494155 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.494014    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cc80bb1-12af-4722-b761-c7cf48940444-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:02:52.494155 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.494027    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3031d17b-7f58-4c88-b495-1e08101ac08f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:02:52.494155 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.494041    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3031d17b-7f58-4c88-b495-1e08101ac08f-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:02:52.494155 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.494052    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5cv4\" (UniqueName: \"kubernetes.io/projected/4cc80bb1-12af-4722-b761-c7cf48940444-kube-api-access-c5cv4\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:02:52.825151 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.825119    2576 generic.go:356] "Generic (PLEG): container finished" podID="4cc80bb1-12af-4722-b761-c7cf48940444" containerID="1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790" exitCode=0
Jul 01 20:02:52.825326 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.825169    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"
Jul 01 20:02:52.825326 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.825201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" event={"ID":"4cc80bb1-12af-4722-b761-c7cf48940444","Type":"ContainerDied","Data":"1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790"}
Jul 01 20:02:52.825326 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.825263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh" event={"ID":"4cc80bb1-12af-4722-b761-c7cf48940444","Type":"ContainerDied","Data":"07d50756d708d97e27cc571979a8db96d69e22c2bc93ed979822f20f1b83bfe7"}
Jul 01 20:02:52.825326 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.825296    2576 scope.go:117] "RemoveContainer" containerID="2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d"
Jul 01 20:02:52.826895 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.826871    2576 generic.go:356] "Generic (PLEG): container finished" podID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerID="361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa" exitCode=0
Jul 01 20:02:52.827004 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.826921    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"
Jul 01 20:02:52.827004 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.826952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" event={"ID":"3031d17b-7f58-4c88-b495-1e08101ac08f","Type":"ContainerDied","Data":"361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa"}
Jul 01 20:02:52.827091 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.827009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x" event={"ID":"3031d17b-7f58-4c88-b495-1e08101ac08f","Type":"ContainerDied","Data":"00f40ac4af776ca43111212aa21dfcf4959285ee96898c067c12b16a845b4321"}
Jul 01 20:02:52.834385 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.834341    2576 scope.go:117] "RemoveContainer" containerID="1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790"
Jul 01 20:02:52.842083 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.842063    2576 scope.go:117] "RemoveContainer" containerID="2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d"
Jul 01 20:02:52.842324 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:02:52.842305    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d\": container with ID starting with 2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d not found: ID does not exist" containerID="2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d"
Jul 01 20:02:52.842379 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.842331    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d"} err="failed to get container status \"2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d\": rpc error: code = NotFound desc = could not find container \"2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d\": container with ID starting with 2959085848df9e725215036d743b829b10befc8a0a31863f7fe0b699545a5c0d not found: ID does not exist"
Jul 01 20:02:52.842379 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.842347    2576 scope.go:117] "RemoveContainer" containerID="1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790"
Jul 01 20:02:52.842587 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:02:52.842570    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790\": container with ID starting with 1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790 not found: ID does not exist" containerID="1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790"
Jul 01 20:02:52.842623 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.842593    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790"} err="failed to get container status \"1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790\": rpc error: code = NotFound desc = could not find container \"1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790\": container with ID starting with 1b94dc8b7e4c395b10c737f950d6a1e572bcf7355c1d413c519d9a953c0c2790 not found: ID does not exist"
Jul 01 20:02:52.842623 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.842608    2576 scope.go:117] "RemoveContainer" containerID="1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a"
Jul 01 20:02:52.850326 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.850306    2576 scope.go:117] "RemoveContainer" containerID="361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa"
Jul 01 20:02:52.851323 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.851304    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"]
Jul 01 20:02:52.857470 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.857436    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5e51f-predictor-6bcc4458db-cbk6x"]
Jul 01 20:02:52.857859 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.857843    2576 scope.go:117] "RemoveContainer" containerID="1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a"
Jul 01 20:02:52.858098 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:02:52.858081    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a\": container with ID starting with 1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a not found: ID does not exist" containerID="1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a"
Jul 01 20:02:52.858180 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.858106    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a"} err="failed to get container status \"1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a\": rpc error: code = NotFound desc = could not find container \"1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a\": container with ID starting with 1fb3ce0df0f30e705de7f2773a249d3f4d4cca7349b92545b7c6fefea263f73a not found: ID does not exist"
Jul 01 20:02:52.858180 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.858128    2576 scope.go:117] "RemoveContainer" containerID="361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa"
Jul 01 20:02:52.858381 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:02:52.858364    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa\": container with ID starting with 361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa not found: ID does not exist" containerID="361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa"
Jul 01 20:02:52.858422 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.858386    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa"} err="failed to get container status \"361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa\": rpc error: code = NotFound desc = could not find container \"361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa\": container with ID starting with 361a05947d1c0cd643eb85606e8916cc68f8d13dc6f118d85b566830a5d4e3aa not found: ID does not exist"
Jul 01 20:02:52.869006 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.868986    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"]
Jul 01 20:02:52.877571 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:52.874412    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5e51f-predictor-ccbbff6dc-42zwh"]
Jul 01 20:02:53.751503 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:53.751466    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" path="/var/lib/kubelet/pods/3031d17b-7f58-4c88-b495-1e08101ac08f/volumes"
Jul 01 20:02:53.752060 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:02:53.752039    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" path="/var/lib/kubelet/pods/4cc80bb1-12af-4722-b761-c7cf48940444/volumes"
Jul 01 20:06:20.560595 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:06:20.560563    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 20:06:20.571756 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:06:20.571734    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 20:10:13.408444 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.408404    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"]
Jul 01 20:10:13.408971 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.408722    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kube-rbac-proxy" containerID="cri-o://d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d" gracePeriod=30
Jul 01 20:10:13.408971 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.408714    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container" containerID="cri-o://48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f" gracePeriod=30
Jul 01 20:10:13.464816 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.464786    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"]
Jul 01 20:10:13.465066 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.465037    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container" containerID="cri-o://ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523" gracePeriod=30
Jul 01 20:10:13.465305 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.465072    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kube-rbac-proxy" containerID="cri-o://3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec" gracePeriod=30
Jul 01 20:10:13.856529 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.856494    2576 generic.go:356] "Generic (PLEG): container finished" podID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerID="d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d" exitCode=2
Jul 01 20:10:13.856686 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.856567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" event={"ID":"1d7fd533-c9ad-40ec-b822-fe5037faae02","Type":"ContainerDied","Data":"d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d"}
Jul 01 20:10:13.858299 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.858271    2576 generic.go:356] "Generic (PLEG): container finished" podID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerID="3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec" exitCode=2
Jul 01 20:10:13.858405 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:13.858323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" event={"ID":"036888f0-9d4c-4c8f-8d5b-8a9937c823b8","Type":"ContainerDied","Data":"3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec"}
Jul 01 20:10:16.278916 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.278885    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 20:10:16.318472 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.318434    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 20:10:16.434821 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.434794    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtbp8\" (UniqueName: \"kubernetes.io/projected/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-api-access-gtbp8\") pod \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") "
Jul 01 20:10:16.434949 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.434841    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-rbac-proxy-sar-config\") pod \"1d7fd533-c9ad-40ec-b822-fe5037faae02\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") "
Jul 01 20:10:16.434949 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.434875    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls\") pod \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") "
Jul 01 20:10:16.434949 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.434906    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7fd533-c9ad-40ec-b822-fe5037faae02-proxy-tls\") pod \"1d7fd533-c9ad-40ec-b822-fe5037faae02\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") "
Jul 01 20:10:16.434949 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.434923    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-rbac-proxy-sar-config\") pod \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\" (UID: \"036888f0-9d4c-4c8f-8d5b-8a9937c823b8\") "
Jul 01 20:10:16.435176 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.434950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k85g\" (UniqueName: \"kubernetes.io/projected/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-api-access-6k85g\") pod \"1d7fd533-c9ad-40ec-b822-fe5037faae02\" (UID: \"1d7fd533-c9ad-40ec-b822-fe5037faae02\") "
Jul 01 20:10:16.435337 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.435308    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d7fd533-c9ad-40ec-b822-fe5037faae02" (UID: "1d7fd533-c9ad-40ec-b822-fe5037faae02"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:10:16.435337 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.435324    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "036888f0-9d4c-4c8f-8d5b-8a9937c823b8" (UID: "036888f0-9d4c-4c8f-8d5b-8a9937c823b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 20:10:16.436985 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.436961    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-api-access-gtbp8" (OuterVolumeSpecName: "kube-api-access-gtbp8") pod "036888f0-9d4c-4c8f-8d5b-8a9937c823b8" (UID: "036888f0-9d4c-4c8f-8d5b-8a9937c823b8"). InnerVolumeSpecName "kube-api-access-gtbp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:10:16.437083 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.437022    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7fd533-c9ad-40ec-b822-fe5037faae02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d7fd533-c9ad-40ec-b822-fe5037faae02" (UID: "1d7fd533-c9ad-40ec-b822-fe5037faae02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:10:16.437320 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.437299    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-api-access-6k85g" (OuterVolumeSpecName: "kube-api-access-6k85g") pod "1d7fd533-c9ad-40ec-b822-fe5037faae02" (UID: "1d7fd533-c9ad-40ec-b822-fe5037faae02"). InnerVolumeSpecName "kube-api-access-6k85g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 20:10:16.437320 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.437306    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "036888f0-9d4c-4c8f-8d5b-8a9937c823b8" (UID: "036888f0-9d4c-4c8f-8d5b-8a9937c823b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 20:10:16.535660 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.535632    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:10:16.535660 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.535657    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:10:16.535798 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.535671    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d7fd533-c9ad-40ec-b822-fe5037faae02-proxy-tls\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:10:16.535798 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.535684    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:10:16.535798 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.535696    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k85g\" (UniqueName: \"kubernetes.io/projected/1d7fd533-c9ad-40ec-b822-fe5037faae02-kube-api-access-6k85g\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:10:16.535798 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.535710    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtbp8\" (UniqueName: \"kubernetes.io/projected/036888f0-9d4c-4c8f-8d5b-8a9937c823b8-kube-api-access-gtbp8\") on node \"ip-10-0-130-234.ec2.internal\" DevicePath \"\""
Jul 01 20:10:16.874428 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.874396    2576 generic.go:356] "Generic (PLEG): container finished" podID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerID="ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523" exitCode=0
Jul 01 20:10:16.874618 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.874445    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"
Jul 01 20:10:16.874618 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.874483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" event={"ID":"036888f0-9d4c-4c8f-8d5b-8a9937c823b8","Type":"ContainerDied","Data":"ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523"}
Jul 01 20:10:16.874618 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.874545    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw" event={"ID":"036888f0-9d4c-4c8f-8d5b-8a9937c823b8","Type":"ContainerDied","Data":"a2c14eb374b82288bf5872ec8458b29727f4fd70b90c6113ee11437966935cb7"}
Jul 01 20:10:16.874618 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.874584    2576 scope.go:117] "RemoveContainer" containerID="3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec"
Jul 01 20:10:16.876141 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.876119    2576 generic.go:356] "Generic (PLEG): container finished" podID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerID="48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f" exitCode=0
Jul 01 20:10:16.876246 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.876153    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" event={"ID":"1d7fd533-c9ad-40ec-b822-fe5037faae02","Type":"ContainerDied","Data":"48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f"}
Jul 01 20:10:16.876246 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.876188    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"
Jul 01 20:10:16.876246 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.876203    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln" event={"ID":"1d7fd533-c9ad-40ec-b822-fe5037faae02","Type":"ContainerDied","Data":"20f133fc2eaaa8e6741bb1dce56c1a3207512c742f6829f053bacf46c512341d"}
Jul 01 20:10:16.883583 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.883565    2576 scope.go:117] "RemoveContainer" containerID="ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523"
Jul 01 20:10:16.891135 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.891117    2576 scope.go:117] "RemoveContainer" containerID="3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec"
Jul 01 20:10:16.891374 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:10:16.891356    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec\": container with ID starting with 3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec not found: ID does not exist" containerID="3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec"
Jul 01 20:10:16.891413 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.891382    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec"} err="failed to get container status \"3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec\": rpc error: code = NotFound desc = could not find container \"3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec\": container with ID starting with 3afa0f50e1b3a952fb781ca7115126fec28b21fb11d8112b73173b71796e48ec not found: ID does not exist"
Jul 01 20:10:16.891413 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.891398    2576 scope.go:117] "RemoveContainer" containerID="ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523"
Jul 01 20:10:16.891638 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:10:16.891623    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523\": container with ID starting with ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523 not found: ID does not exist" containerID="ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523"
Jul 01 20:10:16.891677 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.891642    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523"} err="failed to get container status \"ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523\": rpc error: code = NotFound desc = could not find container \"ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523\": container with ID starting with ada279a3531d84db8042f17be34bc2460f6de960e7fd7a46d441aedbd7558523 not found: ID does not exist"
Jul 01 20:10:16.891677 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.891656    2576 scope.go:117] "RemoveContainer" containerID="d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d"
Jul 01 20:10:16.900444 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.900415    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"]
Jul 01 20:10:16.902164 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.902141    2576 scope.go:117] "RemoveContainer" containerID="48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f"
Jul 01 20:10:16.902875 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.902852    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-00068-predictor-589d75fd87-jdckw"]
Jul 01 20:10:16.909387 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.909369    2576 scope.go:117] "RemoveContainer" containerID="d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d"
Jul 01 20:10:16.909649 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:10:16.909629    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d\": container with ID starting with d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d not found: ID does not exist" containerID="d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d"
Jul 01 20:10:16.909740 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.909658    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d"} err="failed to get container status \"d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d\": rpc error: code = NotFound desc = could not find container \"d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d\": container with ID starting with d2aa12939f446eaa440c1fd620353fc74a89618b7ad56574973f4e6ece35e13d not found: ID does not exist"
Jul 01 20:10:16.909740 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.909678    2576 scope.go:117] "RemoveContainer" containerID="48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f"
Jul 01 20:10:16.909914 ip-10-0-130-234 kubenswrapper[2576]: E0701 20:10:16.909896    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f\": container with ID starting with 48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f not found: ID does not exist" containerID="48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f"
Jul 01 20:10:16.909957 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.909920    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f"} err="failed to get container status \"48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f\": rpc error: code = NotFound desc = could not find container \"48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f\": container with ID starting with 48dd4745e6484634889f1653df8df4cc27c77cc90c60481d600e40f622aa300f not found: ID does not exist"
Jul 01 20:10:16.913288 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.913264    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"]
Jul 01 20:10:16.915012 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:16.914990    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-00068-predictor-cd7ff766-4rlln"]
Jul 01 20:10:17.750817 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:17.750784    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" path="/var/lib/kubelet/pods/036888f0-9d4c-4c8f-8d5b-8a9937c823b8/volumes"
Jul 01 20:10:17.751193 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:17.751178    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" path="/var/lib/kubelet/pods/1d7fd533-c9ad-40ec-b822-fe5037faae02/volumes"
Jul 01 20:10:45.795729 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:45.795653    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nfh7p_c8d1dd1d-8578-45a0-8f2a-cecb9128bb34/global-pull-secret-syncer/0.log"
Jul 01 20:10:45.952595 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:45.952566    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vxzzn_ae9c3578-a195-4a37-9ff9-a3ce0ea5c5a6/konnectivity-agent/0.log"
Jul 01 20:10:45.971548 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:45.971523    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-234.ec2.internal_24c23ff45096c6e80b4ac6b382d5677d/haproxy/0.log"
Jul 01 20:10:50.026666 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.026637    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68580feb-5129-4e3a-aa78-3c5a8a0eb2d6/alertmanager/0.log"
Jul 01 20:10:50.056997 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.056976    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68580feb-5129-4e3a-aa78-3c5a8a0eb2d6/config-reloader/0.log"
Jul 01 20:10:50.082944 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.082918    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68580feb-5129-4e3a-aa78-3c5a8a0eb2d6/kube-rbac-proxy-web/0.log"
Jul 01 20:10:50.107502 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.107477    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68580feb-5129-4e3a-aa78-3c5a8a0eb2d6/kube-rbac-proxy/0.log"
Jul 01 20:10:50.140592 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.140564    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68580feb-5129-4e3a-aa78-3c5a8a0eb2d6/kube-rbac-proxy-metric/0.log"
Jul 01 20:10:50.163881 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.163855    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68580feb-5129-4e3a-aa78-3c5a8a0eb2d6/prom-label-proxy/0.log"
Jul 01 20:10:50.190904 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.190877    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_68580feb-5129-4e3a-aa78-3c5a8a0eb2d6/init-config-reloader/0.log"
Jul 01 20:10:50.236013 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.235948    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-65jxg_4dbf254d-765f-48a8-941f-febf6e272995/cluster-monitoring-operator/0.log"
Jul 01 20:10:50.354376 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.354344    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-557c58445c-mqsq8_766e71e7-dc42-4dd1-bdd7-ff86d19e3804/metrics-server/0.log"
Jul 01 20:10:50.597160 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.597128    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcrv8_a1cf8740-ae32-468d-a9ab-12ec40132eec/node-exporter/0.log"
Jul 01 20:10:50.619174 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.619153    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcrv8_a1cf8740-ae32-468d-a9ab-12ec40132eec/kube-rbac-proxy/0.log"
Jul 01 20:10:50.644056 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.644037    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rcrv8_a1cf8740-ae32-468d-a9ab-12ec40132eec/init-textfile/0.log"
Jul 01 20:10:50.953461 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.953375    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-pzprc_b125ad98-8dc8-4196-b516-0246c7b1e8f9/prometheus-operator/0.log"
Jul 01 20:10:50.978501 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:50.978480    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-pzprc_b125ad98-8dc8-4196-b516-0246c7b1e8f9/kube-rbac-proxy/0.log"
Jul 01 20:10:51.039696 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:51.039673    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7988dcbcdb-6hhd4_ee4a96eb-1b3d-480e-a0ff-596170396e71/telemeter-client/0.log"
Jul 01 20:10:51.065574 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:51.065551    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7988dcbcdb-6hhd4_ee4a96eb-1b3d-480e-a0ff-596170396e71/reload/0.log"
Jul 01 20:10:51.086702 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:51.086687    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7988dcbcdb-6hhd4_ee4a96eb-1b3d-480e-a0ff-596170396e71/kube-rbac-proxy/0.log"
Jul 01 20:10:52.504467 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:52.504413    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-xcw24_1a117e56-ec3c-4923-97b3-a0425bcdae19/networking-console-plugin/0.log"
Jul 01 20:10:53.013978 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:53.013945    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/2.log"
Jul 01 20:10:53.017673 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:53.017641    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7fdbdf9c99-pcrgm_a3b20d7f-4331-4dd8-88ed-bac6347f02e8/console-operator/3.log"
Jul 01 20:10:53.473860 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:53.473832    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-tq7xb_5891a7de-d1fd-4fb1-99cb-473574216100/download-server/0.log"
Jul 01 20:10:53.973523 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:53.973497    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-q2bft_793deba2-438e-4029-a2bb-efdb8460962c/volume-data-source-validator/0.log"
Jul 01 20:10:54.151063 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151034    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"]
Jul 01 20:10:54.151383 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151372    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container"
Jul 01 20:10:54.151427 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151385    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container"
Jul 01 20:10:54.151427 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151394    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container"
Jul 01 20:10:54.151427 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151400    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container"
Jul 01 20:10:54.151427 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151408    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151427 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151414    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151427 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151420    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container"
Jul 01 20:10:54.151427 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151425    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151436    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151442    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151461    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151467    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151472    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151477    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151485    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151490    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151496    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151501    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151507    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151513    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151520    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151526    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151536    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151541    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151592    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151599    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151606    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d7fd533-c9ad-40ec-b822-fe5037faae02" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151615    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151623    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151632    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151638    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a57e3829-7446-4fe1-865f-6b6745092bb7" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151644    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="036888f0-9d4c-4c8f-8d5b-8a9937c823b8" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151653    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151661    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1255d853-0fd5-4a41-9546-ddd0cabb2554" containerName="kube-rbac-proxy"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151668    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3031d17b-7f58-4c88-b495-1e08101ac08f" containerName="kserve-container"
Jul 01 20:10:54.151739 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.151674    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc80bb1-12af-4722-b761-c7cf48940444" containerName="kserve-container"
Jul 01 20:10:54.155006 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.154986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.157411 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.157390    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2sjbp\"/\"kube-root-ca.crt\""
Jul 01 20:10:54.157551 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.157494    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2sjbp\"/\"default-dockercfg-m98cx\""
Jul 01 20:10:54.157791 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.157777    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2sjbp\"/\"openshift-service-ca.crt\""
Jul 01 20:10:54.163260 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.163239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"]
Jul 01 20:10:54.219040 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.219015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6h5\" (UniqueName: \"kubernetes.io/projected/15b23942-4134-4397-898a-82dcb93f74c2-kube-api-access-fg6h5\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.219164 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.219046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-proc\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.219164 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.219148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-sys\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.219277 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.219179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-lib-modules\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.219277 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.219204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-podres\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.319730 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-sys\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.319860 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-lib-modules\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.319860 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-podres\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.319860 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319823    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-sys\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.319860 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6h5\" (UniqueName: \"kubernetes.io/projected/15b23942-4134-4397-898a-82dcb93f74c2-kube-api-access-fg6h5\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.320057 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319874    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-lib-modules\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.320057 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-proc\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.320057 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319928    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-podres\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.320057 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.319981    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15b23942-4134-4397-898a-82dcb93f74c2-proc\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.328054 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.328033    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6h5\" (UniqueName: \"kubernetes.io/projected/15b23942-4134-4397-898a-82dcb93f74c2-kube-api-access-fg6h5\") pod \"perf-node-gather-daemonset-zltsz\" (UID: \"15b23942-4134-4397-898a-82dcb93f74c2\") " pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.464601 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.464579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:54.593771 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.593747    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"]
Jul 01 20:10:54.595615 ip-10-0-130-234 kubenswrapper[2576]: W0701 20:10:54.595589    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod15b23942_4134_4397_898a_82dcb93f74c2.slice/crio-22a4177c2e7807dc16db7a2b9bc922d72671050322fc815b96f3d2e01074c81b WatchSource:0}: Error finding container 22a4177c2e7807dc16db7a2b9bc922d72671050322fc815b96f3d2e01074c81b: Status 404 returned error can't find the container with id 22a4177c2e7807dc16db7a2b9bc922d72671050322fc815b96f3d2e01074c81b
Jul 01 20:10:54.904298 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.904238    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ch6qk_0d95f075-ee0e-498d-8cb0-52a0d3080e5c/dns/0.log"
Jul 01 20:10:54.925602 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:54.925580    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ch6qk_0d95f075-ee0e-498d-8cb0-52a0d3080e5c/kube-rbac-proxy/0.log"
Jul 01 20:10:55.056985 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:55.056955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz" event={"ID":"15b23942-4134-4397-898a-82dcb93f74c2","Type":"ContainerStarted","Data":"b9a3465090cc408dbe77ccfb87cf15a43aa63c95daf2ba9ae046750e94d69e14"}
Jul 01 20:10:55.057326 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:55.057024    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz" event={"ID":"15b23942-4134-4397-898a-82dcb93f74c2","Type":"ContainerStarted","Data":"22a4177c2e7807dc16db7a2b9bc922d72671050322fc815b96f3d2e01074c81b"}
Jul 01 20:10:55.057326 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:55.057102    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:10:55.064321 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:55.064300    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mg6xr_cb7ee79e-263e-4f31-a115-9374f163a704/dns-node-resolver/0.log"
Jul 01 20:10:55.073259 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:55.073217    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz" podStartSLOduration=1.073205608 podStartE2EDuration="1.073205608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 20:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 20:10:55.072010654 +0000 UTC m=+2981.831537376" watchObservedRunningTime="2026-07-01 20:10:55.073205608 +0000 UTC m=+2981.832732331"
Jul 01 20:10:55.614402 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:55.614379    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gc59t_6d88f24b-68e6-4d58-ac3e-98f1c68a55be/node-ca/0.log"
Jul 01 20:10:56.430137 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:56.430112    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75cbd74cc9-tgr8g_cec1ec01-8cca-4ab5-a680-749611f0acb2/router/0.log"
Jul 01 20:10:56.809137 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:56.809116    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2mnld_b55a5e16-7871-451f-a5c6-be4314ca7fb5/serve-healthcheck-canary/0.log"
Jul 01 20:10:57.210904 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:57.210831    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-47xvm_dc491396-37b5-4232-ac52-26beccb0bd39/insights-operator/0.log"
Jul 01 20:10:57.211185 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:57.211168    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-47xvm_dc491396-37b5-4232-ac52-26beccb0bd39/insights-operator/1.log"
Jul 01 20:10:57.342374 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:57.342324    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5nxr_cf0ccf0b-f984-4bcd-8046-8e11b82d19e0/kube-rbac-proxy/0.log"
Jul 01 20:10:57.366994 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:57.366965    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5nxr_cf0ccf0b-f984-4bcd-8046-8e11b82d19e0/exporter/0.log"
Jul 01 20:10:57.391713 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:57.391686    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m5nxr_cf0ccf0b-f984-4bcd-8046-8e11b82d19e0/extractor/0.log"
Jul 01 20:10:59.633535 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:59.633511    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-frxzh_16af8517-1253-40fa-8ac5-65130ddf87d0/server/0.log"
Jul 01 20:10:59.720241 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:59.720209    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-cvgz7_74f96d07-dc86-4757-9bc4-556fb2a00d66/manager/0.log"
Jul 01 20:10:59.793159 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:10:59.793115    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-sj2m2_c693859a-f156-4a43-b3f7-4ccbfdaae776/seaweedfs/0.log"
Jul 01 20:11:01.070526 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:01.070499    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2sjbp/perf-node-gather-daemonset-zltsz"
Jul 01 20:11:04.070886 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:04.070852    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-5fhht_d03eca50-fabd-4cd0-9a1c-465c4ee290c0/kube-storage-version-migrator-operator/1.log"
Jul 01 20:11:04.072003 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:04.071984    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-79964ff6f9-5fhht_d03eca50-fabd-4cd0-9a1c-465c4ee290c0/kube-storage-version-migrator-operator/0.log"
Jul 01 20:11:04.991584 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:04.991558    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hs59_14f2a6f4-c9eb-4462-b413-3fef82d07445/kube-multus-additional-cni-plugins/0.log"
Jul 01 20:11:05.013835 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.013606    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hs59_14f2a6f4-c9eb-4462-b413-3fef82d07445/egress-router-binary-copy/0.log"
Jul 01 20:11:05.035346 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.035328    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hs59_14f2a6f4-c9eb-4462-b413-3fef82d07445/cni-plugins/0.log"
Jul 01 20:11:05.054623 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.054607    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hs59_14f2a6f4-c9eb-4462-b413-3fef82d07445/bond-cni-plugin/0.log"
Jul 01 20:11:05.074348 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.074331    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hs59_14f2a6f4-c9eb-4462-b413-3fef82d07445/routeoverride-cni/0.log"
Jul 01 20:11:05.094530 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.094508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hs59_14f2a6f4-c9eb-4462-b413-3fef82d07445/whereabouts-cni-bincopy/0.log"
Jul 01 20:11:05.113831 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.113810    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7hs59_14f2a6f4-c9eb-4462-b413-3fef82d07445/whereabouts-cni/0.log"
Jul 01 20:11:05.474979 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.474949    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-brnls_6ef721b2-130c-4f92-824b-5b9039cee361/kube-multus/0.log"
Jul 01 20:11:05.619686 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.619662    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r2hbh_a22a2075-444a-40ee-8406-7cd45cd55403/network-metrics-daemon/0.log"
Jul 01 20:11:05.642641 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:05.642621    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r2hbh_a22a2075-444a-40ee-8406-7cd45cd55403/kube-rbac-proxy/0.log"
Jul 01 20:11:06.749702 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.749666    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/ovn-controller/0.log"
Jul 01 20:11:06.766126 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.766101    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/ovn-acl-logging/0.log"
Jul 01 20:11:06.789518 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.789493    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/kube-rbac-proxy-node/0.log"
Jul 01 20:11:06.811164 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.811142    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 20:11:06.846827 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.846803    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/northd/0.log"
Jul 01 20:11:06.867930 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.867908    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/nbdb/0.log"
Jul 01 20:11:06.889097 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.889081    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/sbdb/0.log"
Jul 01 20:11:06.936362 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:06.936341    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sxlfd_a5710820-7283-4fcf-b3a5-b0291c8f274f/ovnkube-controller/0.log"
Jul 01 20:11:07.921805 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:07.921779    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-56499946bc-pcww5_c3ea7a4d-5ae6-4cf4-9d1a-b4ff03c28b21/check-endpoints/0.log"
Jul 01 20:11:07.994794 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:07.994772    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q4qsh_1e5dcab4-31fb-4c7f-a51e-5dc3a0792059/network-check-target-container/0.log"
Jul 01 20:11:08.865912 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:08.865880    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-bf587_af0058fb-09cb-4671-b2a3-1143bb567f59/iptables-alerter/0.log"
Jul 01 20:11:09.566009 ip-10-0-130-234 kubenswrapper[2576]: I0701 20:11:09.565970    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-nbqv4_90f6fad6-42f0-46ba-b45d-0cc2c95c5c73/tuned/0.log"