Aug 04 03:09:35.193544 ip-10-0-130-28 systemd[1]: Starting Kubernetes Kubelet... Aug 04 03:09:37.151099 ip-10-0-130-28 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 03:09:37.151099 ip-10-0-130-28 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 03:09:37.151099 ip-10-0-130-28 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 03:09:37.151099 ip-10-0-130-28 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 03:09:37.151099 ip-10-0-130-28 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 03:09:37.152815 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.152725 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 03:09:37.156855 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156836 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 03:09:37.156855 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156856 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156866 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156871 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156875 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156878 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156881 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156884 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156887 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156902 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156905 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156907 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156910 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156913 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156916 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156918 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156921 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156924 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156926 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156935 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156939 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 03:09:37.156942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156943 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156947 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156950 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156954 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156957 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156960 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156963 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156965 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156968 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156971 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156973 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156976 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156979 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156981 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156984 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156986 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156989 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156992 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156994 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156997 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 03:09:37.157432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.156999 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157002 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157005 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157007 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157010 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157012 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157015 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157017 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157020 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157023 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157026 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157028 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157030 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157034 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157039 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157043 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157046 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157049 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157052 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 03:09:37.158048 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157055 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157057 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157060 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157062 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157065 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157067 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157070 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157073 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157075 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157078 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157082 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157086 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157090 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157092 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157095 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157098 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157101 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157103 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157106 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 03:09:37.158846 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157108 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157111 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157113 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157116 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157119 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157122 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157124 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157127 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157131 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157134 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157136 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157140 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157143 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157145 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157148 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157150 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157153 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157155 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157158 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.157160 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 03:09:37.159583 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158281 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158300 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158305 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158310 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158314 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158318 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158328 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158333 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158338 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158343 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158347 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158352 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158356 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158361 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158365 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158370 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158374 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158380 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158389 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158394 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 03:09:37.160432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158398 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158403 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158408 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158412 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158417 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158421 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158426 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158430 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158435 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158439 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158449 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158453 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158457 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158463 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158468 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158472 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158476 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158480 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158484 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 03:09:37.161335 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158489 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158493 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158497 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158501 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158510 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158514 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158518 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158522 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158527 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158531 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158535 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158540 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158544 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158550 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158555 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158559 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158568 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158572 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158577 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158581 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 03:09:37.162056 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158585 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158592 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158598 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158602 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158607 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158612 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158617 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158622 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158626 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158636 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158640 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158644 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158649 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158656 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158661 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158670 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158675 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158680 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158684 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 03:09:37.162623 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158691 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158702 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158708 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158713 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158717 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158723 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158727 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158731 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158735 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158739 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158745 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158750 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158754 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158759 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158767 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158772 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158776 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158780 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158784 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158793 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 03:09:37.163225 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.158797 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159695 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159712 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159718 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159724 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159728 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159733 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159737 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159742 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159746 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159750 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159754 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159759 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159763 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159767 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159771 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159776 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159780 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159784 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159789 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159793 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 03:09:37.163911 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159798 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159802 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159806 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159810 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159814 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159819 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159823 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159827 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159832 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159836 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159840 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159844 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159848 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159854 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159858 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159863 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159867 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159871 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159876 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 03:09:37.164432 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159880 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159885 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159906 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159912 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159916 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159920 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159924 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159929 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159934 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159941 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159947 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159952 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159957 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159964 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159971 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159977 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159982 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159986 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159991 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 03:09:37.165012 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.159996 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160000 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160004 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160008 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160013 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160017 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160024 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160034 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160039 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160043 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160048 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160053 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160057 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160061 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160066 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160070 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160074 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160078 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160082 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 03:09:37.165492 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160086 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160090 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160095 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160100 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160105 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160110 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160115 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160119 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160123 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160127 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160132 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160136 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160141 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160145 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160149 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160153 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160158 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160162 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160166 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160170 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 03:09:37.165992 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160178 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.160182 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160295 2568 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160306 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160315 2568 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160323 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160330 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160335 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160347 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160353 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160359 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160364 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160370 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160376 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160381 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160386 2568 flags.go:64] FLAG: --cgroup-root="" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160391 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160395 2568 flags.go:64] FLAG: --client-ca-file="" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160401 2568 flags.go:64] FLAG: --cloud-provider="external" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160407 2568 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160421 2568 flags.go:64] FLAG: --cluster-domain="" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160426 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160431 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160436 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 03:09:37.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160441 2568 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160447 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160451 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160456 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160461 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160466 2568 flags.go:64] FLAG: --contention-profiling="false" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160471 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160476 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160484 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160489 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160495 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160500 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160505 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160510 2568 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160515 2568 flags.go:64] FLAG: --enable-server="true" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160520 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160531 2568 flags.go:64] FLAG: --event-burst="100" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160536 2568 flags.go:64] FLAG: --event-qps="50" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160541 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160545 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160550 2568 flags.go:64] FLAG: --eviction-hard="" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160556 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160561 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160566 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160571 2568 flags.go:64] FLAG: --eviction-soft="" Aug 04 03:09:37.167092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160576 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160580 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160590 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160596 2568 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160601 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160606 2568 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160611 2568 flags.go:64] FLAG: --feature-gates="" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160617 2568 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160622 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160627 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160632 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160638 2568 flags.go:64] FLAG: --healthz-port="10248" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160643 2568 flags.go:64] FLAG: --help="false" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160648 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160653 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160662 2568 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160667 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160672 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160678 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160683 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160688 2568 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160693 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160698 2568 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 03:09:37.167721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160703 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160708 2568 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160713 2568 flags.go:64] FLAG: --kube-reserved="" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160718 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160722 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160728 2568 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160732 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160737 2568 flags.go:64] FLAG: --lock-file="" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160742 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160746 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160751 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160761 2568 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160766 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160771 2568 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160776 2568 flags.go:64] FLAG: --logging-format="text" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160781 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160786 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160791 2568 flags.go:64] FLAG: --manifest-url="" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160795 2568 flags.go:64] FLAG: --manifest-url-header="" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160802 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160807 2568 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160815 2568 flags.go:64] FLAG: --max-pods="110" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160820 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160825 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160833 2568 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 03:09:37.168359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160837 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160842 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160847 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160852 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160877 2568 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160882 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160887 2568 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160909 2568 flags.go:64] FLAG: --pod-cidr="" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160914 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160923 2568 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160928 2568 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160933 2568 flags.go:64] FLAG: --pods-per-core="0" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160938 2568 flags.go:64] FLAG: --port="10250" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160943 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160948 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-032d96927762dcdab" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160954 2568 flags.go:64] FLAG: --qos-reserved="" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160959 2568 flags.go:64] FLAG: --read-only-port="10255" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160964 2568 flags.go:64] FLAG: --register-node="true" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160969 2568 flags.go:64] FLAG: --register-with-taints="" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160974 2568 flags.go:64] FLAG: --registry-burst="10" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160980 2568 flags.go:64] FLAG: --registry-qps="5" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160985 2568 flags.go:64] FLAG: --reserved-cpus="" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160990 2568 flags.go:64] FLAG: --reserved-memory="" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.160997 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161002 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 03:09:37.169043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161007 2568 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161013 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161019 2568 flags.go:64] FLAG: --runonce="false" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161024 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161029 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161034 2568 flags.go:64] FLAG: --seccomp-default="false" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161040 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161047 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161052 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161057 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161062 2568 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161067 2568 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161072 2568 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161077 2568 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161081 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161086 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161091 2568 flags.go:64] FLAG: --system-cgroups="" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161096 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161105 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161110 2568 flags.go:64] FLAG: --tls-cert-file="" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161115 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161122 2568 flags.go:64] FLAG: --tls-min-version="" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161127 2568 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161131 2568 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161136 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 03:09:37.169658 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161141 2568 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161146 2568 flags.go:64] FLAG: --v="2" Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161152 2568 flags.go:64] FLAG: --version="false" Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161160 2568 flags.go:64] FLAG: --vmodule="" Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161166 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.161171 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161349 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161355 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161363 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161369 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161374 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161379 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161383 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161389 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161394 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161398 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161402 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161407 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161411 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161416 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161420 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 03:09:37.170328 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161424 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161429 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161433 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161437 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161441 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161445 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161449 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161454 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161458 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161462 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161466 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161471 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161475 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161479 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161484 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161489 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161493 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161498 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161502 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161507 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 03:09:37.170841 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161511 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161516 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161520 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161524 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161530 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161535 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161539 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161544 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161548 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161552 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161557 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161561 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161565 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161570 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161574 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161578 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161583 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161587 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161591 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161596 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 03:09:37.171357 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161600 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161604 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161608 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161613 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161617 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161621 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161625 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161630 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161636 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161640 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161644 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161648 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161653 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161657 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161661 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161665 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161672 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161676 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161680 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 03:09:37.171854 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161685 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161691 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161697 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161701 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161705 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161710 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161714 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161719 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161723 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161727 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161732 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161736 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161741 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161744 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161749 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161753 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161757 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161762 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161769 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 03:09:37.172337 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161775 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161779 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161784 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161790 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161794 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.161798 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.162336 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.169366 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.169382 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.170163 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.170171 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170228 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170233 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170237 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170241 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 03:09:37.172803 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170245 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170248 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170250 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170253 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170257 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170260 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170263 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170266 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170268 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170272 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170275 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170278 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170281 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170285 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170289 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170292 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170295 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170298 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170301 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 03:09:37.173278 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170304 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170306 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170309 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170312 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170314 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170317 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170323 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170326 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170328 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170331 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170334 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170337 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170340 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170343 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170346 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170349 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170352 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170355 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170357 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170360 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 03:09:37.173751 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170363 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170365 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170368 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170370 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170373 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170376 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170378 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170381 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170383 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170385 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170388 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170391 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170394 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170396 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170399 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170401 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170404 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170406 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170410 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170414 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 03:09:37.174252 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170418 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170421 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170424 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170427 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170430 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170432 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170435 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170438 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170440 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170443 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170445 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170448 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170451 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170454 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170457 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170460 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170464 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170467 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170469 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170472 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 03:09:37.174778 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170475 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170477 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170480 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170482 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170486 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170490 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170493 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170496 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170499 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170502 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170506 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170509 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170512 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170515 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170518 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 03:09:37.175288 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170520 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.170525 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170646 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170651 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170656 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170659 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170663 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170666 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170669 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170671 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170674 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170676 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170680 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170684 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 03:09:37.175672 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170687 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170689 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170692 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170695 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170698 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170701 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170704 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170707 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170710 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170713 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170715 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170718 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170721 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170724 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170727 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170730 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170733 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170736 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170739 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170741 2568 feature_gate.go:328] unrecognized feature gate: Example Aug 04 03:09:37.176029 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170744 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170746 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170749 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170752 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170754 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170757 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170760 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170762 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170765 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170767 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170770 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170772 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170775 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170778 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170780 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170783 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170785 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170788 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170790 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170793 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 03:09:37.176521 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170795 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170798 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170800 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170803 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170806 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170808 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170811 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170814 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170817 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170820 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170823 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170826 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170828 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170831 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170834 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170836 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170839 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170842 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170845 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170847 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 03:09:37.177145 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170850 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170853 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170856 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170858 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170861 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170863 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170866 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170869 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170871 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170874 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170876 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170879 2568 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170882 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170885 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170889 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170908 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170911 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170914 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170917 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 03:09:37.177638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170920 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170922 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170925 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170929 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170932 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170935 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170937 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:37.170940 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.170945 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.171077 2568 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.173086 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.174182 2568 server.go:1019] "Starting client certificate rotation" Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.174286 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 03:09:37.178202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.174355 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 03:09:37.178523 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.176595 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 03:09:37.178523 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.177763 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 03:09:37.188479 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.188456 2568 log.go:25] "Validated CRI v1 runtime API" Aug 04 03:09:37.190830 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.190811 2568 log.go:25] "Validated CRI v1 image API" Aug 04 03:09:37.192729 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.192709 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 03:09:37.195406 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.195387 2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 85a3657c-ec9b-4591-983c-2bf344eb9a36:/dev/nvme0n1p4 f9794df4-627c-4242-a1a5-1d627c75ecd3:/dev/nvme0n1p3] Aug 04 03:09:37.195466 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.195406 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 03:09:37.200473 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.200345 2568 manager.go:216] Machine: {Timestamp:2026-08-04 03:09:37.199120731 +0000 UTC m=+0.320190075 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101277 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8d01832a2cf0d05e594b7c6bbef9 SystemUUID:ec2b8d01-832a-2cf0-d05e-594b7c6bbef9 BootID:e6358450-f4b6-4a81-920b-9876e507c958 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b3:5b:8f:20:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b3:5b:8f:20:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:b6:4b:c3:0b:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 03:09:37.200473 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.200465 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 03:09:37.200616 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.200558 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 03:09:37.203317 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.203279 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 03:09:37.203502 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.203316 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 03:09:37.204031 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.204018 2568 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 03:09:37.204088 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.204035 2568 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 03:09:37.204088 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.204053 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 03:09:37.204648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.204636 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 03:09:37.205100 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.205077 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 03:09:37.205456 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.205442 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 03:09:37.209340 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.209324 2568 state_mem.go:36] "Initialized new in-memory state store" Aug 04 03:09:37.209483 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.209472 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 03:09:37.209548 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.209510 2568 kubelet.go:486] "Attempting to sync node with API server" Aug 04 03:09:37.209548 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.209525 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 03:09:37.209548 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.209544 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 03:09:37.209671 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.209559 2568 kubelet.go:398] "Adding apiserver pod source" Aug 04 03:09:37.209671 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.209573 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 03:09:37.210093 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.210076 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 03:09:37.210169 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.210110 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 03:09:37.212248 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.212198 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 03:09:37.213819 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.213798 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 03:09:37.213925 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.213840 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 03:09:37.215252 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215235 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 03:09:37.215326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215257 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 03:09:37.215326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215266 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 03:09:37.215326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215274 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 03:09:37.215326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215288 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 03:09:37.215326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215296 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 03:09:37.215326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215310 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 03:09:37.215326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215318 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 03:09:37.215545 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215343 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 03:09:37.215545 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215353 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 03:09:37.215545 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215365 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 03:09:37.215545 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.215373 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 03:09:37.216129 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.216118 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 03:09:37.216185 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.216132 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 03:09:37.218230 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.218196 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 03:09:37.218373 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.218350 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 03:09:37.218606 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.218573 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 03:09:37.219989 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.219973 2568 server.go:1268] "Started kubelet" Aug 04 03:09:37.220070 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.220048 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 03:09:37.220179 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.220123 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 03:09:37.220219 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.220208 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 03:09:37.220915 ip-10-0-130-28 systemd[1]: Started Kubernetes Kubelet. Aug 04 03:09:37.221539 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.221462 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 03:09:37.225249 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.225060 2568 server.go:310] "Adding debug handlers to kubelet server" Aug 04 03:09:37.226962 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.226940 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 03:09:37.227852 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.227832 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vqckd" Aug 04 03:09:37.229308 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.227489 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18c87b70ee792748 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-08-04 03:09:37.219872584 +0000 UTC m=+0.340941917,LastTimestamp:2026-08-04 03:09:37.219872584 +0000 UTC m=+0.340941917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}" Aug 04 03:09:37.229718 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.229705 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 03:09:37.229832 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.229816 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.229934 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.229842 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 03:09:37.230277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230254 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 03:09:37.230277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230265 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 03:09:37.230277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230273 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 03:09:37.230464 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230441 2568 reconstruct.go:97] "Volume reconstruction finished" Aug 04 03:09:37.230464 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230450 2568 reconciler.go:29] "Reconciler: start to sync state" Aug 04 03:09:37.230721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230706 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 03:09:37.230804 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230723 2568 factory.go:55] Registering systemd factory Aug 04 03:09:37.230804 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230732 2568 factory.go:223] Registration of the systemd container factory successfully Aug 04 03:09:37.231014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.230998 2568 factory.go:153] Registering CRI-O factory Aug 04 03:09:37.231084 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.231035 2568 factory.go:223] Registration of the crio container factory successfully Aug 04 03:09:37.231084 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.231059 2568 factory.go:103] Registering Raw factory Aug 04 03:09:37.231178 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.231094 2568 manager.go:1196] Started watching for new ooms in manager Aug 04 03:09:37.231600 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.231574 2568 manager.go:349] Starting recovery of all containers Aug 04 03:09:37.233507 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.233482 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vqckd" Aug 04 03:09:37.240121 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.240087 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 03:09:37.240258 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.240210 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 03:09:37.243140 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.243123 2568 manager.go:354] Recovery completed Aug 04 03:09:37.247491 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.247478 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 03:09:37.251644 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.251617 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Aug 04 03:09:37.251756 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.251656 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 03:09:37.251756 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.251668 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Aug 04 03:09:37.252315 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.252301 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 03:09:37.252315 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.252311 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 03:09:37.252412 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.252312 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 03:09:37.252412 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.252370 2568 state_mem.go:36] "Initialized new in-memory state store" Aug 04 03:09:37.253580 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.253568 2568 policy_none.go:49] "None policy: Start" Aug 04 03:09:37.253629 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.253584 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 03:09:37.253629 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.253594 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 03:09:37.254633 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.254621 2568 policy_none.go:47] "Start" Aug 04 03:09:37.255195 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.255179 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Aug 04 03:09:37.255283 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.255212 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 03:09:37.255283 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.255228 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Aug 04 03:09:37.291645 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.291492 2568 manager.go:337] "Starting Device Plugin manager" Aug 04 03:09:37.291770 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.291665 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 03:09:37.291770 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.291678 2568 server.go:87] "Starting device plugin registration server" Aug 04 03:09:37.292019 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.292005 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 03:09:37.292087 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.292021 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 03:09:37.292219 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.292204 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 03:09:37.293413 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.292928 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 03:09:37.293413 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.292943 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 03:09:37.293784 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.293766 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 03:09:37.293848 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.293830 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.365292 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.365250 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 03:09:37.366770 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.366505 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 03:09:37.366770 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.366541 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 03:09:37.366770 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.366562 2568 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 03:09:37.366770 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.366606 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 03:09:37.369192 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.369173 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 03:09:37.392358 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.392264 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 03:09:37.394487 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.394467 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Aug 04 03:09:37.394584 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.394501 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 03:09:37.394584 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.394512 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Aug 04 03:09:37.394584 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.394535 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.402798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.402782 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.402863 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.402806 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-28.ec2.internal\": node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.421356 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.421336 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.467164 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.467130 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"] Aug 04 03:09:37.467330 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.467207 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 03:09:37.468946 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.468930 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Aug 04 03:09:37.469011 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.468960 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 03:09:37.469011 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.468970 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Aug 04 03:09:37.470210 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.470199 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 03:09:37.470315 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.470302 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.470362 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.470345 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 03:09:37.471011 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.470996 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Aug 04 03:09:37.471086 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.471022 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 03:09:37.471086 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.471032 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Aug 04 03:09:37.471494 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.471478 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Aug 04 03:09:37.471576 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.471508 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 03:09:37.471576 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.471520 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Aug 04 03:09:37.472753 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.472733 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.472753 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.472757 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 03:09:37.473636 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.473618 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory" Aug 04 03:09:37.473732 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.473648 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 03:09:37.473732 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.473659 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID" Aug 04 03:09:37.496853 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.496829 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.501334 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.501316 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.521961 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.521927 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.532320 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.532286 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd8bacfae2b1c96a838e97cd988a77be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"fd8bacfae2b1c96a838e97cd988a77be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.532456 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.532324 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3c3aa494c6c9dbc7b758da8edbb9441-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"c3c3aa494c6c9dbc7b758da8edbb9441\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.532456 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.532350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3c3aa494c6c9dbc7b758da8edbb9441-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"c3c3aa494c6c9dbc7b758da8edbb9441\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.623076 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.623040 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.633490 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.633465 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3c3aa494c6c9dbc7b758da8edbb9441-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"c3c3aa494c6c9dbc7b758da8edbb9441\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.633612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.633505 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3c3aa494c6c9dbc7b758da8edbb9441-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"c3c3aa494c6c9dbc7b758da8edbb9441\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.633612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.633530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd8bacfae2b1c96a838e97cd988a77be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"fd8bacfae2b1c96a838e97cd988a77be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.633612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.633548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3c3aa494c6c9dbc7b758da8edbb9441-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"c3c3aa494c6c9dbc7b758da8edbb9441\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.633612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.633577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd8bacfae2b1c96a838e97cd988a77be-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"fd8bacfae2b1c96a838e97cd988a77be\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.633747 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.633612 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3c3aa494c6c9dbc7b758da8edbb9441-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"c3c3aa494c6c9dbc7b758da8edbb9441\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.723959 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.723913 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.799566 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.799528 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.804589 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:37.804571 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Aug 04 03:09:37.824094 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.824053 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:37.924666 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:37.924626 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:38.025232 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:38.025140 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:38.125686 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:38.125645 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:38.174137 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.174103 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 03:09:38.174587 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.174263 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 03:09:38.226029 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:38.226007 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:38.233055 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.233031 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 03:09:38.239270 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.239234 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 03:04:37 +0000 UTC" deadline="2028-04-08 06:16:48.93572812 +0000 UTC" Aug 04 03:09:38.239270 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.239264 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14715h7m10.696467309s" Aug 04 03:09:38.269997 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.269969 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 03:09:38.323732 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:38.323699 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8bacfae2b1c96a838e97cd988a77be.slice/crio-a2a97ee85d052e465bf4359a02fc3b2fb024e4bcc5735e5fead7514da90ed75d WatchSource:0}: Error finding container a2a97ee85d052e465bf4359a02fc3b2fb024e4bcc5735e5fead7514da90ed75d: Status 404 returned error can't find the container with id a2a97ee85d052e465bf4359a02fc3b2fb024e4bcc5735e5fead7514da90ed75d Aug 04 03:09:38.324040 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:38.324021 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c3aa494c6c9dbc7b758da8edbb9441.slice/crio-599994b2ee43a538ae16748bd14392464047b5a2fd5f0c3c10804ce828946e48 WatchSource:0}: Error finding container 599994b2ee43a538ae16748bd14392464047b5a2fd5f0c3c10804ce828946e48: Status 404 returned error can't find the container with id 599994b2ee43a538ae16748bd14392464047b5a2fd5f0c3c10804ce828946e48 Aug 04 03:09:38.326125 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:38.326100 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found" Aug 04 03:09:38.328327 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.328309 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 03:09:38.352907 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.352857 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 03:09:38.370290 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.370217 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"c3c3aa494c6c9dbc7b758da8edbb9441","Type":"ContainerStarted","Data":"599994b2ee43a538ae16748bd14392464047b5a2fd5f0c3c10804ce828946e48"} Aug 04 03:09:38.371250 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.371231 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"fd8bacfae2b1c96a838e97cd988a77be","Type":"ContainerStarted","Data":"a2a97ee85d052e465bf4359a02fc3b2fb024e4bcc5735e5fead7514da90ed75d"} Aug 04 03:09:38.430618 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.430593 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" Aug 04 03:09:38.442981 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.442960 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 03:09:38.445190 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.445177 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" Aug 04 03:09:38.453883 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:38.453861 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 03:09:39.211757 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.211515 2568 apiserver.go:52] "Watching apiserver" Aug 04 03:09:39.219697 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.219669 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 03:09:39.220268 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.220246 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-dgxjp","openshift-network-operator/iptables-alerter-f8tsv","openshift-ovn-kubernetes/ovnkube-node-wrchf","kube-system/global-pull-secret-syncer-4k95v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6","openshift-cluster-node-tuning-operator/tuned-5vjlq","openshift-dns/node-resolver-vcc28","openshift-multus/multus-7dpjj","openshift-multus/multus-additional-cni-plugins-4v9nw","kube-system/konnectivity-agent-tslnt","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal","openshift-image-registry/node-ca-9vmv7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","openshift-multus/network-metrics-daemon-lgtdn"] Aug 04 03:09:39.221839 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.221816 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.223652 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.223629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.224403 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.224383 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 03:09:39.224403 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.224396 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5dskl\"" Aug 04 03:09:39.224547 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.224405 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 03:09:39.225122 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.225066 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.228020 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.227995 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 03:09:39.228126 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.228103 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 03:09:39.228348 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.228008 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpfc8\"" Aug 04 03:09:39.228451 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.228431 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 03:09:39.228823 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.228803 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 03:09:39.229119 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.229103 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 03:09:39.230806 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.230020 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 03:09:39.230806 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.230109 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 03:09:39.231343 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.231321 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rrlb7\"" Aug 04 03:09:39.231491 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.231474 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 03:09:39.231627 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.231614 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 03:09:39.231758 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.231742 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.231958 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.231926 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:39.232117 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.232096 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 03:09:39.232117 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.232110 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 03:09:39.234585 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.234562 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.236322 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.236294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.236679 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.236661 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 03:09:39.237094 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.237075 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 03:09:39.237312 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.237292 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 03:09:39.237389 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.237311 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k4jxk\"" Aug 04 03:09:39.238770 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.238728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.239778 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.239485 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 03:09:39.239778 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.239499 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 03:09:39.239778 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.239708 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bqgvj\"" Aug 04 03:09:39.241067 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.240370 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:39.241067 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.240431 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:39.241067 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.240795 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 03:09:39.241264 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241217 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2a765b1-015e-490f-8423-5779334c52a7-hosts-file\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.241264 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241249 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-cni-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.241366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-etc-kubernetes\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.241366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-kubelet\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241326 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241544 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241383 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-env-overrides\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241544 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-ovnkube-script-lib\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241544 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/379b75b7-a52e-4a10-be3d-89794b750ada-kubelet-config\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.241544 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241455 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-sys-fs\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.241544 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241479 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-os-release\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.241544 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-cni-multus\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.241544 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241533 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-cni-netd\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-ovnkube-config\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/193bb2da-bf93-40fc-b40d-e922661ab941-ovn-node-metrics-cert\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6s99\" (UniqueName: \"kubernetes.io/projected/193bb2da-bf93-40fc-b40d-e922661ab941-kube-api-access-j6s99\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241702 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdf6b\" (UniqueName: \"kubernetes.io/projected/29601142-1bdd-4c75-9dd8-414db36a25f4-kube-api-access-wdf6b\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkdf\" (UniqueName: \"kubernetes.io/projected/b2a765b1-015e-490f-8423-5779334c52a7-kube-api-access-6nkdf\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241747 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-cni-binary-copy\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-socket-dir-parent\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241793 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-cni-bin\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/379b75b7-a52e-4a10-be3d-89794b750ada-dbus\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241844 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.241865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-system-cni-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241907 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-cnibin\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-netns\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-kubelet\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.241988 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gcwk\" (UniqueName: \"kubernetes.io/projected/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-kube-api-access-2gcwk\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-socket-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-registration-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-k8s-cni-cncf-io\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242102 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-conf-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-daemon-config\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242136 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-systemd-units\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242152 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-slash\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242165 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-run-netns\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242180 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-device-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242194 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a765b1-015e-490f-8423-5779334c52a7-tmp-dir\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-multus-certs\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242227 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-etc-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.242555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242242 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-node-log\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-log-socket\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242257 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-cni-bin\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242286 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242304 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-hostroot\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-systemd\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242332 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-var-lib-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242347 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-ovn\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.243444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.242496 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-brxhp\"" Aug 04 03:09:39.244539 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.244191 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.245358 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.245337 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 03:09:39.245812 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.245792 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.247094 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.247072 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 03:09:39.247173 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.247141 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zzx7d\"" Aug 04 03:09:39.247240 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.247152 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 03:09:39.247457 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.247440 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:39.247540 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.247507 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:39.248286 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.248236 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 03:09:39.248519 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.248503 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 03:09:39.248824 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.248806 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.249366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.249349 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 03:09:39.249585 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.249569 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-87ldg\"" Aug 04 03:09:39.251179 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.251160 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 03:09:39.251179 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.251174 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 03:09:39.251338 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.251317 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lcmcw\"" Aug 04 03:09:39.251455 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.251436 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 03:09:39.261343 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.261313 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7rz6" Aug 04 03:09:39.267582 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.267531 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b7rz6" Aug 04 03:09:39.293341 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.293317 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 03:09:39.331244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.331218 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/379b75b7-a52e-4a10-be3d-89794b750ada-kubelet-config\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-cni-multus\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-cni-netd\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6s99\" (UniqueName: \"kubernetes.io/projected/193bb2da-bf93-40fc-b40d-e922661ab941-kube-api-access-j6s99\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342669 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-modprobe-d\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342665 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/379b75b7-a52e-4a10-be3d-89794b750ada-kubelet-config\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342695 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d490415-a11e-4f2e-a069-7ede858a698c-tmp\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-cni-netd\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkdf\" (UniqueName: \"kubernetes.io/projected/b2a765b1-015e-490f-8423-5779334c52a7-kube-api-access-6nkdf\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342788 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-cni-bin\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-log-socket\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-var-lib-kubelet\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/379b75b7-a52e-4a10-be3d-89794b750ada-dbus\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342934 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-cni-bin\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.343168 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342946 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-system-cni-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342969 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-netns\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-slash\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.342995 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-cni-multus\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343034 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-slash\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343061 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-log-socket\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2fb4df7-54b7-403d-bcbe-116f661d41c6-serviceca\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343154 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9gkn\" (UniqueName: \"kubernetes.io/projected/382acb1b-3236-4fb9-a499-b4dda99ba2f1-kube-api-access-j9gkn\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-socket-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-registration-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343221 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/379b75b7-a52e-4a10-be3d-89794b750ada-dbus\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-k8s-cni-cncf-io\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343264 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysctl-d\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343272 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-system-cni-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-systemd\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343375 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-registration-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.344078 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.343388 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.343461 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret podName:379b75b7-a52e-4a10-be3d-89794b750ada nodeName:}" failed. No retries permitted until 2026-08-04 03:09:39.843440012 +0000 UTC m=+2.964509353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret") pod "global-pull-secret-syncer-4k95v" (UID: "379b75b7-a52e-4a10-be3d-89794b750ada") : object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-socket-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343545 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-netns\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cnibin\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-device-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343644 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-k8s-cni-cncf-io\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a765b1-015e-490f-8423-5779334c52a7-tmp-dir\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343677 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-device-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-multus-certs\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/382acb1b-3236-4fb9-a499-b4dda99ba2f1-host-slash\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-run-multus-certs\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-hostroot\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343864 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-host\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c824696-e224-4701-a396-936afae4333d-konnectivity-ca\") pod \"konnectivity-agent-tslnt\" (UID: \"6c824696-e224-4701-a396-936afae4333d\") " pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343917 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-hostroot\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2a765b1-015e-490f-8423-5779334c52a7-hosts-file\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.344869 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343969 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-cni-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343975 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a765b1-015e-490f-8423-5779334c52a7-tmp-dir\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.343982 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2a765b1-015e-490f-8423-5779334c52a7-hosts-file\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-kubelet\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344026 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-cni-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344086 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-env-overrides\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344121 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-kubelet\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344208 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysctl-conf\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvpt\" (UniqueName: \"kubernetes.io/projected/ed8629cd-8988-40bb-a461-2155943874ca-kube-api-access-lcvpt\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344557 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-sys-fs\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-os-release\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/193bb2da-bf93-40fc-b40d-e922661ab941-ovn-node-metrics-cert\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-lib-modules\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344651 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-env-overrides\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.345703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-os-release\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-kubernetes\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344740 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-os-release\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344777 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-sys-fs\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d490415-a11e-4f2e-a069-7ede858a698c-etc-tuned\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-ovnkube-config\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.344941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdf6b\" (UniqueName: \"kubernetes.io/projected/29601142-1bdd-4c75-9dd8-414db36a25f4-kube-api-access-wdf6b\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-cni-binary-copy\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345088 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345076 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-socket-dir-parent\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345345 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-systemd\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-ovn\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.346526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-cnibin\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-kubelet\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345477 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gcwk\" (UniqueName: \"kubernetes.io/projected/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-kube-api-access-2gcwk\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345477 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-ovnkube-config\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-systemd-units\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345505 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-conf-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345522 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-cni-binary-copy\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345540 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-conf-dir\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345572 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-host-var-lib-kubelet\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345593 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-cnibin\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345592 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/29601142-1bdd-4c75-9dd8-414db36a25f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345615 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-daemon-config\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345639 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-socket-dir-parent\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-run-netns\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-cni-bin\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345680 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-systemd\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345697 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-ovn\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.347398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345722 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-systemd-units\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345722 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmqwq\" (UniqueName: \"kubernetes.io/projected/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-kube-api-access-zmqwq\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345744 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-run-netns\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345749 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2fb4df7-54b7-403d-bcbe-116f661d41c6-host\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345752 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-cni-bin\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsrn\" (UniqueName: \"kubernetes.io/projected/3d490415-a11e-4f2e-a069-7ede858a698c-kube-api-access-6nsrn\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-system-cni-dir\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345888 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-etc-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345928 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-node-log\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345944 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-etc-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345990 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.345994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-sys\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346033 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-node-log\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/382acb1b-3236-4fb9-a499-b4dda99ba2f1-iptables-alerter-script\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-var-lib-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6clr\" (UniqueName: \"kubernetes.io/projected/e2fb4df7-54b7-403d-bcbe-116f661d41c6-kube-api-access-r6clr\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.348277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysconfig\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-run\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c824696-e224-4701-a396-936afae4333d-agent-certs\") pod \"konnectivity-agent-tslnt\" (UID: \"6c824696-e224-4701-a396-936afae4333d\") " pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346114 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-var-lib-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346189 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-multus-daemon-config\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-etc-kubernetes\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346257 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-etc-kubernetes\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-ovnkube-script-lib\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346367 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/193bb2da-bf93-40fc-b40d-e922661ab941-run-openvswitch\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.349106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.346746 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/193bb2da-bf93-40fc-b40d-e922661ab941-ovnkube-script-lib\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.350203 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.349577 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/193bb2da-bf93-40fc-b40d-e922661ab941-ovn-node-metrics-cert\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.351696 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.351675 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkdf\" (UniqueName: \"kubernetes.io/projected/b2a765b1-015e-490f-8423-5779334c52a7-kube-api-access-6nkdf\") pod \"node-resolver-vcc28\" (UID: \"b2a765b1-015e-490f-8423-5779334c52a7\") " pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.352477 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.352457 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6s99\" (UniqueName: \"kubernetes.io/projected/193bb2da-bf93-40fc-b40d-e922661ab941-kube-api-access-j6s99\") pod \"ovnkube-node-wrchf\" (UID: \"193bb2da-bf93-40fc-b40d-e922661ab941\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.352804 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.352781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gcwk\" (UniqueName: \"kubernetes.io/projected/cee1d932-e8b1-4374-8d76-9c98e6c35d7a-kube-api-access-2gcwk\") pod \"multus-7dpjj\" (UID: \"cee1d932-e8b1-4374-8d76-9c98e6c35d7a\") " pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.354425 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.354381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdf6b\" (UniqueName: \"kubernetes.io/projected/29601142-1bdd-4c75-9dd8-414db36a25f4-kube-api-access-wdf6b\") pod \"aws-ebs-csi-driver-node-z9wh6\" (UID: \"29601142-1bdd-4c75-9dd8-414db36a25f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.447238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.447238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2fb4df7-54b7-403d-bcbe-116f661d41c6-serviceca\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.447478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447432 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.447478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9gkn\" (UniqueName: \"kubernetes.io/projected/382acb1b-3236-4fb9-a499-b4dda99ba2f1-kube-api-access-j9gkn\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.447573 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysctl-d\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.447573 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-systemd\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.447573 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cnibin\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.447680 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447589 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/382acb1b-3236-4fb9-a499-b4dda99ba2f1-host-slash\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.447680 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-host\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.447680 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c824696-e224-4701-a396-936afae4333d-konnectivity-ca\") pod \"konnectivity-agent-tslnt\" (UID: \"6c824696-e224-4701-a396-936afae4333d\") " pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.447680 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:39.447798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447682 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2fb4df7-54b7-403d-bcbe-116f661d41c6-serviceca\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.447798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysctl-d\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.447798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-systemd\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.447798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447724 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/382acb1b-3236-4fb9-a499-b4dda99ba2f1-host-slash\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.447798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447765 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cnibin\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.447798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447769 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysctl-conf\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447800 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvpt\" (UniqueName: \"kubernetes.io/projected/ed8629cd-8988-40bb-a461-2155943874ca-kube-api-access-lcvpt\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-lib-modules\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-os-release\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-kubernetes\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d490415-a11e-4f2e-a069-7ede858a698c-etc-tuned\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447926 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysctl-conf\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447928 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447962 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.447999 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:39.448014 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmqwq\" (UniqueName: \"kubernetes.io/projected/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-kube-api-access-zmqwq\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.448051 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:09:39.948032416 +0000 UTC m=+3.069101762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448059 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-lib-modules\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2fb4df7-54b7-403d-bcbe-116f661d41c6-host\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsrn\" (UniqueName: \"kubernetes.io/projected/3d490415-a11e-4f2e-a069-7ede858a698c-kube-api-access-6nsrn\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448139 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-system-cni-dir\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.447803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-host\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448164 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-sys\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448168 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6c824696-e224-4701-a396-936afae4333d-konnectivity-ca\") pod \"konnectivity-agent-tslnt\" (UID: \"6c824696-e224-4701-a396-936afae4333d\") " pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/382acb1b-3236-4fb9-a499-b4dda99ba2f1-iptables-alerter-script\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6clr\" (UniqueName: \"kubernetes.io/projected/e2fb4df7-54b7-403d-bcbe-116f661d41c6-kube-api-access-r6clr\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-sys\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysconfig\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448238 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-os-release\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-run\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c824696-e224-4701-a396-936afae4333d-agent-certs\") pod \"konnectivity-agent-tslnt\" (UID: \"6c824696-e224-4701-a396-936afae4333d\") " pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.448478 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-modprobe-d\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d490415-a11e-4f2e-a069-7ede858a698c-tmp\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-var-lib-kubelet\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2fb4df7-54b7-403d-bcbe-116f661d41c6-host\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448515 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-var-lib-kubelet\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448664 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-system-cni-dir\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448773 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cni-binary-copy\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-modprobe-d\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.449074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-sysconfig\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.449132 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-run\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.448274 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d490415-a11e-4f2e-a069-7ede858a698c-etc-kubernetes\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.449295 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.449148 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.449912 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.449626 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/382acb1b-3236-4fb9-a499-b4dda99ba2f1-iptables-alerter-script\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.449912 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.449630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.451500 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.451480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d490415-a11e-4f2e-a069-7ede858a698c-etc-tuned\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.452078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.452054 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d490415-a11e-4f2e-a069-7ede858a698c-tmp\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.452986 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.452813 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6c824696-e224-4701-a396-936afae4333d-agent-certs\") pod \"konnectivity-agent-tslnt\" (UID: \"6c824696-e224-4701-a396-936afae4333d\") " pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.454322 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.454269 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 03:09:39.454322 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.454292 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 03:09:39.454322 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.454308 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xlh6v for pod openshift-network-diagnostics/network-check-target-dgxjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:39.454529 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.454440 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v podName:8862a995-e389-475f-b17a-69916d161490 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:39.954409461 +0000 UTC m=+3.075478805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xlh6v" (UniqueName: "kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v") pod "network-check-target-dgxjp" (UID: "8862a995-e389-475f-b17a-69916d161490") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:39.457152 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.457066 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmqwq\" (UniqueName: \"kubernetes.io/projected/5ff0d620-1bad-46a0-afcd-7cc8bcddccbc-kube-api-access-zmqwq\") pod \"multus-additional-cni-plugins-4v9nw\" (UID: \"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc\") " pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.457579 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.457474 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6clr\" (UniqueName: \"kubernetes.io/projected/e2fb4df7-54b7-403d-bcbe-116f661d41c6-kube-api-access-r6clr\") pod \"node-ca-9vmv7\" (UID: \"e2fb4df7-54b7-403d-bcbe-116f661d41c6\") " pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.457806 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.457786 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsrn\" (UniqueName: \"kubernetes.io/projected/3d490415-a11e-4f2e-a069-7ede858a698c-kube-api-access-6nsrn\") pod \"tuned-5vjlq\" (UID: \"3d490415-a11e-4f2e-a069-7ede858a698c\") " pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.458152 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.458111 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9gkn\" (UniqueName: \"kubernetes.io/projected/382acb1b-3236-4fb9-a499-b4dda99ba2f1-kube-api-access-j9gkn\") pod \"iptables-alerter-f8tsv\" (UID: \"382acb1b-3236-4fb9-a499-b4dda99ba2f1\") " pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.459080 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.459059 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvpt\" (UniqueName: \"kubernetes.io/projected/ed8629cd-8988-40bb-a461-2155943874ca-kube-api-access-lcvpt\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:39.537134 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.537044 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcc28" Aug 04 03:09:39.547017 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.546989 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7dpjj" Aug 04 03:09:39.558736 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.558716 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:09:39.563612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.563590 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" Aug 04 03:09:39.574212 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.574147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" Aug 04 03:09:39.582193 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.582165 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" Aug 04 03:09:39.589993 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.589970 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8tsv" Aug 04 03:09:39.598750 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.598713 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:09:39.605714 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.605683 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9vmv7" Aug 04 03:09:39.852082 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.852001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:39.852235 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.852169 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:39.852290 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.852239 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret podName:379b75b7-a52e-4a10-be3d-89794b750ada nodeName:}" failed. No retries permitted until 2026-08-04 03:09:40.85221905 +0000 UTC m=+3.973288375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret") pod "global-pull-secret-syncer-4k95v" (UID: "379b75b7-a52e-4a10-be3d-89794b750ada") : object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:39.953066 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:39.953031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:39.953226 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.953162 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:39.953226 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:39.953216 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:09:40.953200393 +0000 UTC m=+4.074269716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:40.053462 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:40.053431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:40.053643 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.053600 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 03:09:40.053643 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.053617 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 03:09:40.053643 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.053626 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xlh6v for pod openshift-network-diagnostics/network-check-target-dgxjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:40.053745 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.053676 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v podName:8862a995-e389-475f-b17a-69916d161490 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:41.053661192 +0000 UTC m=+4.174730511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlh6v" (UniqueName: "kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v") pod "network-check-target-dgxjp" (UID: "8862a995-e389-475f-b17a-69916d161490") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:40.270010 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:40.269962 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 03:04:39 +0000 UTC" deadline="2028-04-30 04:02:59.734442568 +0000 UTC" Aug 04 03:09:40.270010 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:40.270007 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15240h53m19.464440935s" Aug 04 03:09:40.378079 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.378045 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d490415_a11e_4f2e_a069_7ede858a698c.slice/crio-01f388f2724023b5a62181cb922086af3a91a89754fc1bc6af9775c8a605b400 WatchSource:0}: Error finding container 01f388f2724023b5a62181cb922086af3a91a89754fc1bc6af9775c8a605b400: Status 404 returned error can't find the container with id 01f388f2724023b5a62181cb922086af3a91a89754fc1bc6af9775c8a605b400 Aug 04 03:09:40.390540 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.390486 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382acb1b_3236_4fb9_a499_b4dda99ba2f1.slice/crio-033e8d0d9bb49a6a52f1107fa2687794c239f408c21e60c1502b8617af16e349 WatchSource:0}: Error finding container 033e8d0d9bb49a6a52f1107fa2687794c239f408c21e60c1502b8617af16e349: Status 404 returned error can't find the container with id 033e8d0d9bb49a6a52f1107fa2687794c239f408c21e60c1502b8617af16e349 Aug 04 03:09:40.392543 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.392514 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29601142_1bdd_4c75_9dd8_414db36a25f4.slice/crio-6f4d4c46ed9a7f148e8224b49847d721e27ae94aea9ebed773d7f54ea2a03b26 WatchSource:0}: Error finding container 6f4d4c46ed9a7f148e8224b49847d721e27ae94aea9ebed773d7f54ea2a03b26: Status 404 returned error can't find the container with id 6f4d4c46ed9a7f148e8224b49847d721e27ae94aea9ebed773d7f54ea2a03b26 Aug 04 03:09:40.393248 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.393138 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193bb2da_bf93_40fc_b40d_e922661ab941.slice/crio-452fac29f78d1dea87cfdc20491d2ac2f68964ced1b205fc4e2fc6261e98761b WatchSource:0}: Error finding container 452fac29f78d1dea87cfdc20491d2ac2f68964ced1b205fc4e2fc6261e98761b: Status 404 returned error can't find the container with id 452fac29f78d1dea87cfdc20491d2ac2f68964ced1b205fc4e2fc6261e98761b Aug 04 03:09:40.394686 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.394663 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a765b1_015e_490f_8423_5779334c52a7.slice/crio-0b0f1104bbabb9f8477cb4e5817f53ab897a3f59ab96efba0925ccc4601f2ea7 WatchSource:0}: Error finding container 0b0f1104bbabb9f8477cb4e5817f53ab897a3f59ab96efba0925ccc4601f2ea7: Status 404 returned error can't find the container with id 0b0f1104bbabb9f8477cb4e5817f53ab897a3f59ab96efba0925ccc4601f2ea7 Aug 04 03:09:40.397141 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.396779 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c824696_e224_4701_a396_936afae4333d.slice/crio-7a89d217a9982a05ea5ea6b1449546b2878a2aba5abb03a5f49d1cdade2905e9 WatchSource:0}: Error finding container 7a89d217a9982a05ea5ea6b1449546b2878a2aba5abb03a5f49d1cdade2905e9: Status 404 returned error can't find the container with id 7a89d217a9982a05ea5ea6b1449546b2878a2aba5abb03a5f49d1cdade2905e9 Aug 04 03:09:40.397431 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.397409 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fb4df7_54b7_403d_bcbe_116f661d41c6.slice/crio-a6549553de99aa1115f05ab11bd65f2b88874070feaf3cb177a181983b56b72c WatchSource:0}: Error finding container a6549553de99aa1115f05ab11bd65f2b88874070feaf3cb177a181983b56b72c: Status 404 returned error can't find the container with id a6549553de99aa1115f05ab11bd65f2b88874070feaf3cb177a181983b56b72c Aug 04 03:09:40.399417 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.399317 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee1d932_e8b1_4374_8d76_9c98e6c35d7a.slice/crio-230252ac0c4a60739b7474cf5f1019fd9a70e3cb0a55b875f1d0239d3ec0cc95 WatchSource:0}: Error finding container 230252ac0c4a60739b7474cf5f1019fd9a70e3cb0a55b875f1d0239d3ec0cc95: Status 404 returned error can't find the container with id 230252ac0c4a60739b7474cf5f1019fd9a70e3cb0a55b875f1d0239d3ec0cc95 Aug 04 03:09:40.400409 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:09:40.400377 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff0d620_1bad_46a0_afcd_7cc8bcddccbc.slice/crio-f5683c2ae55eeb9cc80b63d7b0aa74186a9f721c1a69d4012d15d77b49d242b9 WatchSource:0}: Error finding container f5683c2ae55eeb9cc80b63d7b0aa74186a9f721c1a69d4012d15d77b49d242b9: Status 404 returned error can't find the container with id f5683c2ae55eeb9cc80b63d7b0aa74186a9f721c1a69d4012d15d77b49d242b9 Aug 04 03:09:40.859964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:40.859283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:40.859964 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.859469 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:40.859964 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.859532 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret podName:379b75b7-a52e-4a10-be3d-89794b750ada nodeName:}" failed. No retries permitted until 2026-08-04 03:09:42.859514531 +0000 UTC m=+5.980583863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret") pod "global-pull-secret-syncer-4k95v" (UID: "379b75b7-a52e-4a10-be3d-89794b750ada") : object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:40.960105 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:40.959909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:40.960276 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.960112 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:40.960276 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:40.960179 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:09:42.96015935 +0000 UTC m=+6.081228676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:41.061272 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.061233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:41.061455 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:41.061408 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 03:09:41.061455 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:41.061432 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 03:09:41.061455 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:41.061446 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xlh6v for pod openshift-network-diagnostics/network-check-target-dgxjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:41.061619 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:41.061505 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v podName:8862a995-e389-475f-b17a-69916d161490 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:43.06148584 +0000 UTC m=+6.182555164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlh6v" (UniqueName: "kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v") pod "network-check-target-dgxjp" (UID: "8862a995-e389-475f-b17a-69916d161490") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:41.270659 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.270586 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 03:04:39 +0000 UTC" deadline="2028-03-29 11:32:06.055891461 +0000 UTC" Aug 04 03:09:41.270659 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.270658 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14480h22m24.78523811s" Aug 04 03:09:41.369928 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.369883 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:41.370112 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:41.370024 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:41.370220 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.370201 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:41.370283 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.370233 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:41.370358 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:41.370335 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:41.370452 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:41.370432 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:41.384586 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.384555 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8tsv" event={"ID":"382acb1b-3236-4fb9-a499-b4dda99ba2f1","Type":"ContainerStarted","Data":"033e8d0d9bb49a6a52f1107fa2687794c239f408c21e60c1502b8617af16e349"} Aug 04 03:09:41.391904 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.391859 2568 generic.go:356] "Generic (PLEG): container finished" podID="c3c3aa494c6c9dbc7b758da8edbb9441" containerID="429c0d17be022ed4949b8425388add98d5b2c718a938a105549866ca6bad1139" exitCode=0 Aug 04 03:09:41.392827 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.392801 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"c3c3aa494c6c9dbc7b758da8edbb9441","Type":"ContainerDied","Data":"429c0d17be022ed4949b8425388add98d5b2c718a938a105549866ca6bad1139"} Aug 04 03:09:41.399925 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.399036 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"fd8bacfae2b1c96a838e97cd988a77be","Type":"ContainerStarted","Data":"ddb40ed46eb24d5a55b2712a90642c88f9f3df5f2814c4c5b39135ef70f06727"} Aug 04 03:09:41.419282 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.419251 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" event={"ID":"29601142-1bdd-4c75-9dd8-414db36a25f4","Type":"ContainerStarted","Data":"6f4d4c46ed9a7f148e8224b49847d721e27ae94aea9ebed773d7f54ea2a03b26"} Aug 04 03:09:41.432993 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.432753 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" event={"ID":"3d490415-a11e-4f2e-a069-7ede858a698c","Type":"ContainerStarted","Data":"01f388f2724023b5a62181cb922086af3a91a89754fc1bc6af9775c8a605b400"} Aug 04 03:09:41.445654 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.445560 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerStarted","Data":"f5683c2ae55eeb9cc80b63d7b0aa74186a9f721c1a69d4012d15d77b49d242b9"} Aug 04 03:09:41.457583 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.457547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7dpjj" event={"ID":"cee1d932-e8b1-4374-8d76-9c98e6c35d7a","Type":"ContainerStarted","Data":"230252ac0c4a60739b7474cf5f1019fd9a70e3cb0a55b875f1d0239d3ec0cc95"} Aug 04 03:09:41.483341 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.483236 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9vmv7" event={"ID":"e2fb4df7-54b7-403d-bcbe-116f661d41c6","Type":"ContainerStarted","Data":"a6549553de99aa1115f05ab11bd65f2b88874070feaf3cb177a181983b56b72c"} Aug 04 03:09:41.498627 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.498474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tslnt" event={"ID":"6c824696-e224-4701-a396-936afae4333d","Type":"ContainerStarted","Data":"7a89d217a9982a05ea5ea6b1449546b2878a2aba5abb03a5f49d1cdade2905e9"} Aug 04 03:09:41.530754 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.530375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcc28" event={"ID":"b2a765b1-015e-490f-8423-5779334c52a7","Type":"ContainerStarted","Data":"0b0f1104bbabb9f8477cb4e5817f53ab897a3f59ab96efba0925ccc4601f2ea7"} Aug 04 03:09:41.538977 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:41.538816 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"452fac29f78d1dea87cfdc20491d2ac2f68964ced1b205fc4e2fc6261e98761b"} Aug 04 03:09:42.550021 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:42.549950 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"c3c3aa494c6c9dbc7b758da8edbb9441","Type":"ContainerStarted","Data":"9dc4ef4d6c7d91ffc5fe4efecb2d3000b3ee73ce24831f7fdd14fc1ef6d72aaf"} Aug 04 03:09:42.568264 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:42.567703 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" podStartSLOduration=4.567682363 podStartE2EDuration="4.567682363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:41.423125599 +0000 UTC m=+4.544194965" watchObservedRunningTime="2026-08-04 03:09:42.567682363 +0000 UTC m=+5.688751707" Aug 04 03:09:42.880351 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:42.879685 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:42.880351 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:42.879880 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:42.880351 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:42.879981 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret podName:379b75b7-a52e-4a10-be3d-89794b750ada nodeName:}" failed. No retries permitted until 2026-08-04 03:09:46.87996076 +0000 UTC m=+10.001030082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret") pod "global-pull-secret-syncer-4k95v" (UID: "379b75b7-a52e-4a10-be3d-89794b750ada") : object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:42.980652 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:42.980617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:42.980833 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:42.980822 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:42.980920 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:42.980881 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:09:46.98086761 +0000 UTC m=+10.101936930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:43.081115 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:43.081077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:43.081331 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:43.081308 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 03:09:43.081414 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:43.081347 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 03:09:43.081414 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:43.081361 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xlh6v for pod openshift-network-diagnostics/network-check-target-dgxjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:43.081515 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:43.081420 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v podName:8862a995-e389-475f-b17a-69916d161490 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:47.081402844 +0000 UTC m=+10.202472178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlh6v" (UniqueName: "kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v") pod "network-check-target-dgxjp" (UID: "8862a995-e389-475f-b17a-69916d161490") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:43.368115 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:43.368076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:43.368281 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:43.368235 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:43.368659 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:43.368638 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:43.368753 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:43.368737 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:43.368922 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:43.368906 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:43.369017 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:43.368999 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:45.369080 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:45.369046 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:45.369451 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:45.369224 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:45.369584 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:45.369567 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:45.369673 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:45.369656 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:45.369717 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:45.369704 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:45.369798 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:45.369781 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:46.914995 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:46.914952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:46.915451 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:46.915120 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:46.915451 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:46.915202 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret podName:379b75b7-a52e-4a10-be3d-89794b750ada nodeName:}" failed. No retries permitted until 2026-08-04 03:09:54.915182121 +0000 UTC m=+18.036251466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret") pod "global-pull-secret-syncer-4k95v" (UID: "379b75b7-a52e-4a10-be3d-89794b750ada") : object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:47.016012 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:47.015974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:47.016578 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.016171 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:47.016578 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.016249 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:09:55.016229751 +0000 UTC m=+18.137299073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:47.116685 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:47.116569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:47.116851 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.116730 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 03:09:47.116851 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.116753 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 03:09:47.116851 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.116766 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xlh6v for pod openshift-network-diagnostics/network-check-target-dgxjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:47.116851 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.116828 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v podName:8862a995-e389-475f-b17a-69916d161490 nodeName:}" failed. No retries permitted until 2026-08-04 03:09:55.116808394 +0000 UTC m=+18.237877713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlh6v" (UniqueName: "kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v") pod "network-check-target-dgxjp" (UID: "8862a995-e389-475f-b17a-69916d161490") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:47.368590 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:47.368052 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:47.368590 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:47.368160 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:47.368590 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.368174 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:47.368590 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.368281 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:47.368590 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:47.368317 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:47.368590 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:47.368434 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:49.367571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:49.367503 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:49.367571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:49.367546 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:49.368101 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:49.367503 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:49.368101 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:49.367639 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:49.368101 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:49.367735 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:49.368101 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:49.367794 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:51.367435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:51.367377 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:51.367914 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:51.367388 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:51.367914 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:51.367540 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:51.368106 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:51.368073 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:51.368173 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:51.368147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:51.368233 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:51.368214 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:53.367738 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:53.367702 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:53.367738 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:53.367729 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:53.368310 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:53.367718 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:53.368310 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:53.367847 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:53.368310 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:53.367970 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:53.368310 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:53.368100 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:54.978584 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:54.978543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:54.979116 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:54.978710 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:54.979116 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:54.978791 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret podName:379b75b7-a52e-4a10-be3d-89794b750ada nodeName:}" failed. No retries permitted until 2026-08-04 03:10:10.97877093 +0000 UTC m=+34.099840253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret") pod "global-pull-secret-syncer-4k95v" (UID: "379b75b7-a52e-4a10-be3d-89794b750ada") : object "kube-system"/"original-pull-secret" not registered Aug 04 03:09:55.079338 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:55.079298 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:55.079537 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.079465 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:55.079602 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.079553 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:10:11.079530218 +0000 UTC m=+34.200599542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:09:55.180503 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:55.180467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:55.180700 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.180604 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 03:09:55.180700 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.180624 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 03:09:55.180700 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.180637 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xlh6v for pod openshift-network-diagnostics/network-check-target-dgxjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:55.180700 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.180696 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v podName:8862a995-e389-475f-b17a-69916d161490 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:11.180680862 +0000 UTC m=+34.301750206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlh6v" (UniqueName: "kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v") pod "network-check-target-dgxjp" (UID: "8862a995-e389-475f-b17a-69916d161490") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:09:55.367346 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:55.367260 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:55.367346 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:55.367282 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:55.367558 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:55.367278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:55.367558 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.367394 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:55.367716 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.367549 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:55.367716 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:55.367692 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:57.367867 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:57.367830 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:57.367867 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:57.367850 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:57.368449 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:57.367969 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:57.368449 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:57.368085 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:57.368449 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:57.368166 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:57.368449 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:57.368277 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:59.367293 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.366951 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:09:59.367974 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.366993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:09:59.367974 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:59.367390 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:09:59.367974 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.366951 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:09:59.367974 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:59.367461 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:09:59.367974 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:09:59.367518 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:09:59.584817 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.584779 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ff0d620-1bad-46a0-afcd-7cc8bcddccbc" containerID="3e69f498c3abba836d4cd64bf08ac1af7160f6b9711c46948534712011b07c39" exitCode=0 Aug 04 03:09:59.584992 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.584907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerDied","Data":"3e69f498c3abba836d4cd64bf08ac1af7160f6b9711c46948534712011b07c39"} Aug 04 03:09:59.586465 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.586437 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7dpjj" event={"ID":"cee1d932-e8b1-4374-8d76-9c98e6c35d7a","Type":"ContainerStarted","Data":"6501a704db500d3f5876c34763dc6bc798bfe56f23c40f34f31c6fd91203686f"} Aug 04 03:09:59.588129 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.588089 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9vmv7" event={"ID":"e2fb4df7-54b7-403d-bcbe-116f661d41c6","Type":"ContainerStarted","Data":"290e6d46e1434f9b1430b9e05fb7aa821bdb40e6ade614b85cd162f29cd4f590"} Aug 04 03:09:59.589607 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.589584 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tslnt" event={"ID":"6c824696-e224-4701-a396-936afae4333d","Type":"ContainerStarted","Data":"41de400ddaf64ce0cc948f06018173fae3cceda1525a5038b519ccad54e05efe"} Aug 04 03:09:59.591182 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.591158 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcc28" event={"ID":"b2a765b1-015e-490f-8423-5779334c52a7","Type":"ContainerStarted","Data":"879c0a92ad037305d872db8be9f7339f1098986209f47523068995b506a52b7e"} Aug 04 03:09:59.594115 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.594094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"aedc29877c0c012c885b27c3b6f571c203abe8a359697a94d184edc819b85315"} Aug 04 03:09:59.594199 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.594164 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"122341b29490c4c57877253b8496cdbe204cbac13047821e839a9aba66f7018f"} Aug 04 03:09:59.594265 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.594216 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"9e9de01ec1f9de78dd743ab5db4c08a0688dc2663b2cb757d37a7433939aea2c"} Aug 04 03:09:59.594316 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.594265 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"bf0eecd9038ae49b36b72a50794f719957a29e8d540c5fa3d9851d2b83587746"} Aug 04 03:09:59.594316 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.594303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"78127f6dfc4fc9ffea56b9065669e9705ee3fa945511c6e9da84f379f8e9840f"} Aug 04 03:09:59.594417 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.594340 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"ae95d010be60668a1798489920aacb5552333e72ba14388f2d5ab78506346f3a"} Aug 04 03:09:59.595487 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.595467 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" event={"ID":"29601142-1bdd-4c75-9dd8-414db36a25f4","Type":"ContainerStarted","Data":"667dce38ccdfe0d6ce430b2b7ecc1c65240b43b2457474e57e4f2006e24f07eb"} Aug 04 03:09:59.596800 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.596780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" event={"ID":"3d490415-a11e-4f2e-a069-7ede858a698c","Type":"ContainerStarted","Data":"fe04ecb4cb9a58ad002b99b9ccd8f687c9aeef795d91a2435c13173c5dc59351"} Aug 04 03:09:59.607096 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.607057 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" podStartSLOduration=21.607042366 podStartE2EDuration="21.607042366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:42.568131127 +0000 UTC m=+5.689200469" watchObservedRunningTime="2026-08-04 03:09:59.607042366 +0000 UTC m=+22.728111708" Aug 04 03:09:59.641573 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.641528 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vcc28" podStartSLOduration=4.29046137 podStartE2EDuration="22.641510551s" totalImagesPullingTime="18.351049181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:59.641322214 +0000 UTC m=+22.762391555" watchObservedRunningTime="2026-08-04 03:09:59.641510551 +0000 UTC m=+22.762579893" Aug 04 03:09:59.641799 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.641775 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9vmv7" podStartSLOduration=4.291726368 podStartE2EDuration="22.641769612s" totalImagesPullingTime="18.350043244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:59.620142077 +0000 UTC m=+22.741211419" watchObservedRunningTime="2026-08-04 03:09:59.641769612 +0000 UTC m=+22.762838954" Aug 04 03:09:59.671880 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.671779 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5vjlq" podStartSLOduration=4.296800443 podStartE2EDuration="22.671764867s" totalImagesPullingTime="18.374964424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:59.671478061 +0000 UTC m=+22.792547415" watchObservedRunningTime="2026-08-04 03:09:59.671764867 +0000 UTC m=+22.792834208" Aug 04 03:09:59.672062 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.671957 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7dpjj" podStartSLOduration=4.270247817 podStartE2EDuration="22.671948324s" totalImagesPullingTime="18.401700507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:59.656840064 +0000 UTC m=+22.777909405" watchObservedRunningTime="2026-08-04 03:09:59.671948324 +0000 UTC m=+22.793017665" Aug 04 03:09:59.685318 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:09:59.685271 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tslnt" podStartSLOduration=12.397344874 podStartE2EDuration="22.685252371s" totalImagesPullingTime="10.287907497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:09:59.684588317 +0000 UTC m=+22.805657659" watchObservedRunningTime="2026-08-04 03:09:59.685252371 +0000 UTC m=+22.806321714" Aug 04 03:10:00.254093 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.254060 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 03:10:00.303534 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.303363 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T03:10:00.254088986Z","UUID":"050be6fb-d673-4879-96f4-05eceb67a882","Handler":null,"Name":"","Endpoint":""} Aug 04 03:10:00.306383 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.306126 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 03:10:00.306383 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.306171 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 03:10:00.557562 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.557521 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:10:00.558097 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.558079 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:10:00.601531 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.601178 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8tsv" event={"ID":"382acb1b-3236-4fb9-a499-b4dda99ba2f1","Type":"ContainerStarted","Data":"0537b7a4c5c8f41960d1715abc9e3bd8012be0b897e401502d0c08a386d00d61"} Aug 04 03:10:00.603542 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.603513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" event={"ID":"29601142-1bdd-4c75-9dd8-414db36a25f4","Type":"ContainerStarted","Data":"9f4f68e1e0749e6c1c6b4f6bbd715c57f501715998c9b77276f1c1337131452c"} Aug 04 03:10:00.604523 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.604493 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:10:00.604633 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.604559 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tslnt" Aug 04 03:10:00.616296 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:00.616246 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f8tsv" podStartSLOduration=5.256779973 podStartE2EDuration="23.616227269s" totalImagesPullingTime="18.359447296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:00.615600222 +0000 UTC m=+23.736669565" watchObservedRunningTime="2026-08-04 03:10:00.616227269 +0000 UTC m=+23.737296612" Aug 04 03:10:01.366999 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:01.366958 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:01.367199 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:01.366958 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:01.367199 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:01.367091 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:10:01.367199 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:01.367173 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:10:01.367199 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:01.366967 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:01.367411 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:01.367250 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:10:01.607990 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:01.607913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" event={"ID":"29601142-1bdd-4c75-9dd8-414db36a25f4","Type":"ContainerStarted","Data":"d34fbf2fed968a123a1b880c6231c84b772b976e86bd7fc7ada712740bd67e0d"} Aug 04 03:10:01.627534 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:01.627432 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z9wh6" podStartSLOduration=3.976365555 podStartE2EDuration="24.6274095s" totalImagesPullingTime="20.651043945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:01.626943007 +0000 UTC m=+24.748012370" watchObservedRunningTime="2026-08-04 03:10:01.6274095 +0000 UTC m=+24.748478843" Aug 04 03:10:02.614328 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:02.614287 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"12c326ac99002369db5e703497be321ff1cb95d114cdd5afcde85797360c492f"} Aug 04 03:10:03.367280 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:03.367037 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:03.367466 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:03.367096 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:03.367466 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:03.367392 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:10:03.367466 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:03.367123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:03.367644 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:03.367466 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:10:03.367644 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:03.367553 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:10:04.623159 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:04.623121 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" event={"ID":"193bb2da-bf93-40fc-b40d-e922661ab941","Type":"ContainerStarted","Data":"f31b0c93b47e405c483104e2c0ee5d92f69fd899e2a916e2590fec14cc43b161"} Aug 04 03:10:04.624104 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:04.623394 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:10:04.625108 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:04.624972 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ff0d620-1bad-46a0-afcd-7cc8bcddccbc" containerID="34bceecac13693b1c8326b8400ab38b2dd2aef0de5fad77ded566f1aaae155a7" exitCode=0 Aug 04 03:10:04.625705 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:04.625050 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerDied","Data":"34bceecac13693b1c8326b8400ab38b2dd2aef0de5fad77ded566f1aaae155a7"} Aug 04 03:10:04.640264 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:04.640242 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:10:04.648907 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:04.648856 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" podStartSLOduration=9.205933515 podStartE2EDuration="27.648834916s" totalImagesPullingTime="18.442901401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:04.647088725 +0000 UTC m=+27.768158081" watchObservedRunningTime="2026-08-04 03:10:04.648834916 +0000 UTC m=+27.769904283" Aug 04 03:10:05.354998 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.354959 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tslnt_6c824696-e224-4701-a396-936afae4333d/konnectivity-agent/0.log" limit=1024 Aug 04 03:10:05.367101 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.367076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:05.367101 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.367090 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:05.367322 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:05.367180 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:10:05.367322 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:05.367250 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:10:05.367322 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.367270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:05.367482 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:05.367360 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:10:05.627871 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.627843 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:10:05.628253 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.627938 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:10:05.645771 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.645741 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:10:05.858039 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.858003 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgxjp"] Aug 04 03:10:05.858214 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.858128 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:05.858281 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:05.858235 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:10:05.861082 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.861052 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4k95v"] Aug 04 03:10:05.861242 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.861144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:05.861302 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:05.861237 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:10:05.861685 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.861659 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lgtdn"] Aug 04 03:10:05.861798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:05.861767 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:05.861877 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:05.861859 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:10:06.631357 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:06.631321 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ff0d620-1bad-46a0-afcd-7cc8bcddccbc" containerID="0dc90364780394a4dc7a0bfca48bf7e15cbaf0a815b63495f68627840b65443c" exitCode=0 Aug 04 03:10:06.631760 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:06.631412 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerDied","Data":"0dc90364780394a4dc7a0bfca48bf7e15cbaf0a815b63495f68627840b65443c"} Aug 04 03:10:07.368110 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:07.368075 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:07.368292 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:07.368164 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:07.368292 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:07.368194 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:10:07.368402 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:07.368285 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:10:07.368402 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:07.368340 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:07.368402 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:07.368394 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:10:08.639125 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:08.638955 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ff0d620-1bad-46a0-afcd-7cc8bcddccbc" containerID="96115a31d937cc4ce846a3e28585091cf7cd3ae47b43a5609bc76ea548a1c42e" exitCode=0 Aug 04 03:10:08.639568 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:08.639022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerDied","Data":"96115a31d937cc4ce846a3e28585091cf7cd3ae47b43a5609bc76ea548a1c42e"} Aug 04 03:10:09.367232 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:09.367197 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:09.367410 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:09.367244 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:09.367410 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:09.367205 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:09.367410 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:09.367335 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:10:09.367410 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:09.367386 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:10:09.367640 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:09.367475 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:10:11.004396 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:11.004357 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:11.004837 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.004533 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 03:10:11.004837 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.004616 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret podName:379b75b7-a52e-4a10-be3d-89794b750ada nodeName:}" failed. No retries permitted until 2026-08-04 03:10:43.00459579 +0000 UTC m=+66.125665113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret") pod "global-pull-secret-syncer-4k95v" (UID: "379b75b7-a52e-4a10-be3d-89794b750ada") : object "kube-system"/"original-pull-secret" not registered Aug 04 03:10:11.105358 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:11.105319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:11.105539 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.105460 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:10:11.105539 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.105529 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:10:43.105511433 +0000 UTC m=+66.226580771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 03:10:11.206187 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:11.206158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:11.206371 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.206352 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 03:10:11.206447 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.206381 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 03:10:11.206447 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.206395 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xlh6v for pod openshift-network-diagnostics/network-check-target-dgxjp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:10:11.206550 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.206459 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v podName:8862a995-e389-475f-b17a-69916d161490 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:43.206440173 +0000 UTC m=+66.327509511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xlh6v" (UniqueName: "kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v") pod "network-check-target-dgxjp" (UID: "8862a995-e389-475f-b17a-69916d161490") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 03:10:11.367784 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:11.367690 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:11.367935 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:11.367691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:11.367988 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.367937 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4k95v" podUID="379b75b7-a52e-4a10-be3d-89794b750ada" Aug 04 03:10:11.367988 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.367833 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:10:11.367988 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:11.367970 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:11.368084 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:11.368014 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dgxjp" podUID="8862a995-e389-475f-b17a-69916d161490" Aug 04 03:10:12.705619 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.705584 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeReady" Aug 04 03:10:12.706171 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.705748 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 03:10:12.747972 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.747914 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tsqgz"] Aug 04 03:10:12.750569 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.750539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:12.750710 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.750643 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mp8nz"] Aug 04 03:10:12.752407 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.752389 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:12.753156 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.752957 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 03:10:12.753156 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.752991 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4d7vw\"" Aug 04 03:10:12.753156 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.753094 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 03:10:12.755048 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.755025 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 03:10:12.755144 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.755031 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xvw8f\"" Aug 04 03:10:12.755196 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.755033 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 03:10:12.755311 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.755282 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 03:10:12.760636 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.760615 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tsqgz"] Aug 04 03:10:12.763206 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.763178 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mp8nz"] Aug 04 03:10:12.918472 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.918430 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7b8\" (UniqueName: \"kubernetes.io/projected/cbe2b01b-b6ca-46f2-a852-c087b167d470-kube-api-access-dw7b8\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:12.918662 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.918504 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8b9811-69f2-4b6a-ab51-1e200e549ff8-cert\") pod \"ingress-canary-mp8nz\" (UID: \"4e8b9811-69f2-4b6a-ab51-1e200e549ff8\") " pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:12.918662 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.918529 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:12.918662 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.918552 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe2b01b-b6ca-46f2-a852-c087b167d470-tmp-dir\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:12.918662 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.918572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4q5\" (UniqueName: \"kubernetes.io/projected/4e8b9811-69f2-4b6a-ab51-1e200e549ff8-kube-api-access-nx4q5\") pod \"ingress-canary-mp8nz\" (UID: \"4e8b9811-69f2-4b6a-ab51-1e200e549ff8\") " pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:12.918816 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:12.918651 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cbe2b01b-b6ca-46f2-a852-c087b167d470-config-volume\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.019882 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.019784 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7b8\" (UniqueName: \"kubernetes.io/projected/cbe2b01b-b6ca-46f2-a852-c087b167d470-kube-api-access-dw7b8\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.019882 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.019859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8b9811-69f2-4b6a-ab51-1e200e549ff8-cert\") pod \"ingress-canary-mp8nz\" (UID: \"4e8b9811-69f2-4b6a-ab51-1e200e549ff8\") " pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:13.020138 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.019887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.020138 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.019930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe2b01b-b6ca-46f2-a852-c087b167d470-tmp-dir\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.020138 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:13.020015 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:10:13.020138 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:13.020088 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:13.520066321 +0000 UTC m=+36.641135654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:10:13.020138 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.020123 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4q5\" (UniqueName: \"kubernetes.io/projected/4e8b9811-69f2-4b6a-ab51-1e200e549ff8-kube-api-access-nx4q5\") pod \"ingress-canary-mp8nz\" (UID: \"4e8b9811-69f2-4b6a-ab51-1e200e549ff8\") " pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:13.020384 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.020160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cbe2b01b-b6ca-46f2-a852-c087b167d470-config-volume\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.020384 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.020272 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbe2b01b-b6ca-46f2-a852-c087b167d470-tmp-dir\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.020776 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.020751 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cbe2b01b-b6ca-46f2-a852-c087b167d470-config-volume\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.025008 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.024982 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8b9811-69f2-4b6a-ab51-1e200e549ff8-cert\") pod \"ingress-canary-mp8nz\" (UID: \"4e8b9811-69f2-4b6a-ab51-1e200e549ff8\") " pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:13.029358 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.029340 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4q5\" (UniqueName: \"kubernetes.io/projected/4e8b9811-69f2-4b6a-ab51-1e200e549ff8-kube-api-access-nx4q5\") pod \"ingress-canary-mp8nz\" (UID: \"4e8b9811-69f2-4b6a-ab51-1e200e549ff8\") " pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:13.029472 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.029451 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7b8\" (UniqueName: \"kubernetes.io/projected/cbe2b01b-b6ca-46f2-a852-c087b167d470-kube-api-access-dw7b8\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.069567 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.069527 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mp8nz" Aug 04 03:10:13.214723 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.214562 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mp8nz"] Aug 04 03:10:13.219809 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:10:13.219782 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8b9811_69f2_4b6a_ab51_1e200e549ff8.slice/crio-7b2a22dcaceb5b77c794631a2ec8d43d9cd3d1382f059ebb101e844670e95cf0 WatchSource:0}: Error finding container 7b2a22dcaceb5b77c794631a2ec8d43d9cd3d1382f059ebb101e844670e95cf0: Status 404 returned error can't find the container with id 7b2a22dcaceb5b77c794631a2ec8d43d9cd3d1382f059ebb101e844670e95cf0 Aug 04 03:10:13.367150 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.367105 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:13.367150 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.367097 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:13.367150 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.367131 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:13.370354 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.370172 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 03:10:13.370354 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.370189 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 03:10:13.370354 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.370235 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jb5ws\"" Aug 04 03:10:13.370354 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.370245 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 03:10:13.371095 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.371080 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6l47c\"" Aug 04 03:10:13.371095 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.371089 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 03:10:13.525236 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.525197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:13.525394 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:13.525351 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:10:13.525431 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:13.525416 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:14.525398567 +0000 UTC m=+37.646467890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:10:13.652314 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.652282 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ff0d620-1bad-46a0-afcd-7cc8bcddccbc" containerID="6e1e3ceeb538bc10e80b2f30599cf9596e30910b458fcd6d21b993570b0ca190" exitCode=0 Aug 04 03:10:13.652472 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.652367 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerDied","Data":"6e1e3ceeb538bc10e80b2f30599cf9596e30910b458fcd6d21b993570b0ca190"} Aug 04 03:10:13.653516 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:13.653493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mp8nz" event={"ID":"4e8b9811-69f2-4b6a-ab51-1e200e549ff8","Type":"ContainerStarted","Data":"7b2a22dcaceb5b77c794631a2ec8d43d9cd3d1382f059ebb101e844670e95cf0"} Aug 04 03:10:14.533414 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:14.533381 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:14.533847 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:14.533497 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:10:14.533847 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:14.533549 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:16.533535837 +0000 UTC m=+39.654605158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:10:14.658642 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:14.658609 2568 generic.go:356] "Generic (PLEG): container finished" podID="5ff0d620-1bad-46a0-afcd-7cc8bcddccbc" containerID="cb48894b1479c25b6820f12321a5085d4a166a6ee603642f30fd6fc00413fe21" exitCode=0 Aug 04 03:10:14.658798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:14.658671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerDied","Data":"cb48894b1479c25b6820f12321a5085d4a166a6ee603642f30fd6fc00413fe21"} Aug 04 03:10:15.665987 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:15.665764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" event={"ID":"5ff0d620-1bad-46a0-afcd-7cc8bcddccbc","Type":"ContainerStarted","Data":"00e5a78531cb3cb90dbb10649ec0c191b0a3b156dcc374efa6b6de48aa7368e5"} Aug 04 03:10:15.691674 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:15.691631 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4v9nw" podStartSLOduration=9.246548013 podStartE2EDuration="38.691617628s" totalImagesPullingTime="29.445069615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:15.690019282 +0000 UTC m=+38.811088644" watchObservedRunningTime="2026-08-04 03:10:15.691617628 +0000 UTC m=+38.812686970" Aug 04 03:10:16.550870 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:16.550831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:16.551029 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:16.550992 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:10:16.551086 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:16.551054 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:20.551039145 +0000 UTC m=+43.672108465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:10:16.670156 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:16.670113 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mp8nz" event={"ID":"4e8b9811-69f2-4b6a-ab51-1e200e549ff8","Type":"ContainerStarted","Data":"8a53310c36e4be6b1e2c851feb1f856b39daf5ab01fc09afe0a8538e05f87608"} Aug 04 03:10:16.685172 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:16.685129 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mp8nz" podStartSLOduration=2.166387303 podStartE2EDuration="4.685112961s" totalImagesPullingTime="2.518725658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:16.684614457 +0000 UTC m=+39.805683819" watchObservedRunningTime="2026-08-04 03:10:16.685112961 +0000 UTC m=+39.806182304" Aug 04 03:10:20.579507 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:20.579462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:20.579920 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:20.579607 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:10:20.579920 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:20.579675 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:28.579659488 +0000 UTC m=+51.700728812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:10:28.627718 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:28.627677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:28.628227 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:28.627821 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:10:28.628227 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:28.627887 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:10:44.627870655 +0000 UTC m=+67.748939975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:10:37.650564 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:37.650528 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrchf" Aug 04 03:10:43.017612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.017573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:43.020402 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.020380 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 03:10:43.032092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.032064 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/379b75b7-a52e-4a10-be3d-89794b750ada-original-pull-secret\") pod \"global-pull-secret-syncer-4k95v\" (UID: \"379b75b7-a52e-4a10-be3d-89794b750ada\") " pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:43.087549 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.087498 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4k95v" Aug 04 03:10:43.118714 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.118676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:10:43.121520 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.121500 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 03:10:43.129680 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:43.129653 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 03:10:43.129792 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:43.129759 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:11:47.129734599 +0000 UTC m=+130.250803931 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : secret "metrics-daemon-secret" not found Aug 04 03:10:43.208043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.208011 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4k95v"] Aug 04 03:10:43.211314 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:10:43.211282 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379b75b7_a52e_4a10_be3d_89794b750ada.slice/crio-d3589aab28a820d659f1c70f1f2bcdda3df90c9fe75e91775c518e169dfa2300 WatchSource:0}: Error finding container d3589aab28a820d659f1c70f1f2bcdda3df90c9fe75e91775c518e169dfa2300: Status 404 returned error can't find the container with id d3589aab28a820d659f1c70f1f2bcdda3df90c9fe75e91775c518e169dfa2300 Aug 04 03:10:43.219236 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.219214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:43.221683 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.221665 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 03:10:43.231730 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.231715 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 03:10:43.242943 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.242921 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlh6v\" (UniqueName: \"kubernetes.io/projected/8862a995-e389-475f-b17a-69916d161490-kube-api-access-xlh6v\") pod \"network-check-target-dgxjp\" (UID: \"8862a995-e389-475f-b17a-69916d161490\") " pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:43.385246 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.385172 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jb5ws\"" Aug 04 03:10:43.393385 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.393364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:43.519698 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.519669 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dgxjp"] Aug 04 03:10:43.522727 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:10:43.522697 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8862a995_e389_475f_b17a_69916d161490.slice/crio-212201f33b23b71a66aa51d43f84a9fb3cdd473afe7c3c537f7e8c1df15ee4ac WatchSource:0}: Error finding container 212201f33b23b71a66aa51d43f84a9fb3cdd473afe7c3c537f7e8c1df15ee4ac: Status 404 returned error can't find the container with id 212201f33b23b71a66aa51d43f84a9fb3cdd473afe7c3c537f7e8c1df15ee4ac Aug 04 03:10:43.732334 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.732297 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgxjp" event={"ID":"8862a995-e389-475f-b17a-69916d161490","Type":"ContainerStarted","Data":"212201f33b23b71a66aa51d43f84a9fb3cdd473afe7c3c537f7e8c1df15ee4ac"} Aug 04 03:10:43.733582 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:43.733554 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4k95v" event={"ID":"379b75b7-a52e-4a10-be3d-89794b750ada","Type":"ContainerStarted","Data":"d3589aab28a820d659f1c70f1f2bcdda3df90c9fe75e91775c518e169dfa2300"} Aug 04 03:10:44.630468 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:44.630431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:10:44.630872 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:44.630573 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:10:44.630872 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:10:44.630627 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:11:16.630613144 +0000 UTC m=+99.751682464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:10:48.748872 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:48.748835 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dgxjp" event={"ID":"8862a995-e389-475f-b17a-69916d161490","Type":"ContainerStarted","Data":"771b15a76854b8b66ef564cda1a4980afba5a85ad781952bc07c7ee5d799ae03"} Aug 04 03:10:48.749341 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:48.748945 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:10:48.750298 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:48.750273 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4k95v" event={"ID":"379b75b7-a52e-4a10-be3d-89794b750ada","Type":"ContainerStarted","Data":"7dc0d635232a224c2183d01a9d1f8744d3102052ce89cf729d46155fb88d9bc8"} Aug 04 03:10:48.764453 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:48.764408 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dgxjp" podStartSLOduration=67.284113408 podStartE2EDuration="1m11.764394928s" totalImagesPullingTime="4.48028152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:48.763844945 +0000 UTC m=+71.884914292" watchObservedRunningTime="2026-08-04 03:10:48.764394928 +0000 UTC m=+71.885464247" Aug 04 03:10:48.778963 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:10:48.778912 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4k95v" podStartSLOduration=66.98315141 podStartE2EDuration="1m11.778881856s" totalImagesPullingTime="4.795730446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:10:48.778019657 +0000 UTC m=+71.899089020" watchObservedRunningTime="2026-08-04 03:10:48.778881856 +0000 UTC m=+71.899951198" Aug 04 03:11:09.096366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.096256 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n"] Aug 04 03:11:09.099112 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.099092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.102903 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.102870 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 03:11:09.102903 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.102884 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 03:11:09.103053 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.102905 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 03:11:09.103053 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.102939 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 04 03:11:09.103053 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.102873 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6rjmb\"" Aug 04 03:11:09.107717 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.107696 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n"] Aug 04 03:11:09.227706 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.227677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8396974f-8891-44ac-aab0-62681a82d0c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n\" (UID: \"8396974f-8891-44ac-aab0-62681a82d0c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.227706 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.227712 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwpjm\" (UniqueName: \"kubernetes.io/projected/8396974f-8891-44ac-aab0-62681a82d0c1-kube-api-access-dwpjm\") pod \"managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n\" (UID: \"8396974f-8891-44ac-aab0-62681a82d0c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.328701 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.328673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8396974f-8891-44ac-aab0-62681a82d0c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n\" (UID: \"8396974f-8891-44ac-aab0-62681a82d0c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.328772 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.328707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwpjm\" (UniqueName: \"kubernetes.io/projected/8396974f-8891-44ac-aab0-62681a82d0c1-kube-api-access-dwpjm\") pod \"managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n\" (UID: \"8396974f-8891-44ac-aab0-62681a82d0c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.331409 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.331383 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8396974f-8891-44ac-aab0-62681a82d0c1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n\" (UID: \"8396974f-8891-44ac-aab0-62681a82d0c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.338753 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.338730 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwpjm\" (UniqueName: \"kubernetes.io/projected/8396974f-8891-44ac-aab0-62681a82d0c1-kube-api-access-dwpjm\") pod \"managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n\" (UID: \"8396974f-8891-44ac-aab0-62681a82d0c1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.419772 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.419740 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" Aug 04 03:11:09.534403 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.534369 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n"] Aug 04 03:11:09.537441 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:11:09.537418 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8396974f_8891_44ac_aab0_62681a82d0c1.slice/crio-ef2d0ea9de4576aa6a11534ea35cd224adcabc0e325f7d924764aa97eecee2a5 WatchSource:0}: Error finding container ef2d0ea9de4576aa6a11534ea35cd224adcabc0e325f7d924764aa97eecee2a5: Status 404 returned error can't find the container with id ef2d0ea9de4576aa6a11534ea35cd224adcabc0e325f7d924764aa97eecee2a5 Aug 04 03:11:09.799505 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:09.799424 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" event={"ID":"8396974f-8891-44ac-aab0-62681a82d0c1","Type":"ContainerStarted","Data":"ef2d0ea9de4576aa6a11534ea35cd224adcabc0e325f7d924764aa97eecee2a5"} Aug 04 03:11:12.808615 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:12.808581 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" event={"ID":"8396974f-8891-44ac-aab0-62681a82d0c1","Type":"ContainerStarted","Data":"5dc80064e7c1bae545cb868559fabf50c9b865692abed520adeee7bc5e9df027"} Aug 04 03:11:12.824119 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:12.824077 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" podStartSLOduration=1.524625683 podStartE2EDuration="3.824061066s" totalImagesPullingTime="2.299435383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:11:12.823635717 +0000 UTC m=+95.944705061" watchObservedRunningTime="2026-08-04 03:11:12.824061066 +0000 UTC m=+95.945130386" Aug 04 03:11:16.679912 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:16.679856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:11:16.680317 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:11:16.679999 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:11:16.680317 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:11:16.680089 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:12:20.680073524 +0000 UTC m=+163.801142844 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:11:19.755989 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:19.755960 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dgxjp" Aug 04 03:11:47.178718 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:11:47.178657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:11:47.179253 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:11:47.178811 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 03:11:47.179253 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:11:47.178889 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:13:49.178871828 +0000 UTC m=+252.299941148 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : secret "metrics-daemon-secret" not found Aug 04 03:12:15.763267 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:12:15.763159 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tsqgz" podUID="cbe2b01b-b6ca-46f2-a852-c087b167d470" Aug 04 03:12:15.959678 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:12:15.959645 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tsqgz" Aug 04 03:12:16.377571 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:12:16.377535 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:12:20.710009 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:12:20.709967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:12:20.710538 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:12:20.710082 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 03:12:20.710538 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:12:20.710147 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls podName:cbe2b01b-b6ca-46f2-a852-c087b167d470 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:22.710132287 +0000 UTC m=+285.831201607 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls") pod "dns-default-tsqgz" (UID: "cbe2b01b-b6ca-46f2-a852-c087b167d470") : secret "dns-default-metrics-tls" not found Aug 04 03:12:30.367549 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:12:30.367465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:13:12.092201 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:12.092167 2568 generic.go:356] "Generic (PLEG): container finished" podID="8396974f-8891-44ac-aab0-62681a82d0c1" containerID="5dc80064e7c1bae545cb868559fabf50c9b865692abed520adeee7bc5e9df027" exitCode=255 Aug 04 03:13:12.092583 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:12.092230 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" event={"ID":"8396974f-8891-44ac-aab0-62681a82d0c1","Type":"ContainerDied","Data":"5dc80064e7c1bae545cb868559fabf50c9b865692abed520adeee7bc5e9df027"} Aug 04 03:13:12.097563 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:12.097545 2568 scope.go:117] "RemoveContainer" containerID="5dc80064e7c1bae545cb868559fabf50c9b865692abed520adeee7bc5e9df027" Aug 04 03:13:13.096160 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:13.096129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f9c54d7b-qcs2n" event={"ID":"8396974f-8891-44ac-aab0-62681a82d0c1","Type":"ContainerStarted","Data":"0759498436fff67d60d0e7db789ecc00270fd6532a79bf58e9904845f7d44844"} Aug 04 03:13:49.187790 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:49.187757 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:13:49.188284 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:49.187870 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 03:13:49.188284 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:49.187937 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs podName:ed8629cd-8988-40bb-a461-2155943874ca nodeName:}" failed. No retries permitted until 2026-08-04 03:15:51.187920715 +0000 UTC m=+374.308990040 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs") pod "network-metrics-daemon-lgtdn" (UID: "ed8629cd-8988-40bb-a461-2155943874ca") : secret "metrics-daemon-secret" not found Aug 04 03:13:51.165212 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.165176 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-885fb9fbb-7xtvr"] Aug 04 03:13:51.168032 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.168017 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.172700 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.172657 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 03:13:51.173128 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.173110 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 03:13:51.173228 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.173165 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9k69h\"" Aug 04 03:13:51.173228 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.173189 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 03:13:51.173228 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.173212 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 03:13:51.174046 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.174028 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 03:13:51.174046 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.174040 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 03:13:51.188082 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.188057 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-885fb9fbb-7xtvr"] Aug 04 03:13:51.259285 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.259259 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-fvfjv"] Aug 04 03:13:51.262055 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.262026 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.263490 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.263469 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h"] Aug 04 03:13:51.264444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.264421 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 03:13:51.264696 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.264680 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 03:13:51.265028 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.265010 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 03:13:51.265348 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.265333 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 03:13:51.266053 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.266039 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7fsrs\"" Aug 04 03:13:51.266332 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.266318 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-hkqrp"] Aug 04 03:13:51.266444 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.266430 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.269146 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.269073 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f8965f5cc-t6xm7"] Aug 04 03:13:51.269218 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.269164 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.270252 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.270233 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 03:13:51.270425 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.270273 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9rlfr\"" Aug 04 03:13:51.270881 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.270285 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 03:13:51.273547 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.270314 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 03:13:51.273547 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.270344 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 03:13:51.273547 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.272307 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 03:13:51.273547 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.272427 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 03:13:51.273766 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.273710 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 03:13:51.274122 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.274099 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 03:13:51.275155 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.275131 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r4qdl\"" Aug 04 03:13:51.276922 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.276868 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 03:13:51.277273 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.277014 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.278042 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.278024 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-fvfjv"] Aug 04 03:13:51.279299 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.279285 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h"] Aug 04 03:13:51.280372 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.280348 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-hkqrp"] Aug 04 03:13:51.280533 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.280512 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 03:13:51.280624 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.280573 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 03:13:51.280742 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.280718 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 03:13:51.280922 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.280884 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9lvw5\"" Aug 04 03:13:51.281010 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.280967 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 03:13:51.286613 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.286594 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 03:13:51.289852 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.289833 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8965f5cc-t6xm7"] Aug 04 03:13:51.300660 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.300642 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-stats-auth\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.300739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.300665 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.300739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.300707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.300739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.300730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-default-certificate\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.300846 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.300763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxtx\" (UniqueName: \"kubernetes.io/projected/42c1c927-755a-4827-99ca-58cd1cb90a46-kube-api-access-cmxtx\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.401610 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.401610 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401612 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-trusted-ca-bundle\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.401800 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13504d71-ed32-4373-b189-966cf1a7f494-trusted-ca\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.401800 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401656 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9f2069c-9b92-451c-ac80-087ecf892a3d-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.401800 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdvgs\" (UniqueName: \"kubernetes.io/projected/b9f2069c-9b92-451c-ac80-087ecf892a3d-kube-api-access-xdvgs\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.401800 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-image-registry-private-configuration\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.401800 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.401751 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:51.901729989 +0000 UTC m=+255.022799312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : configmap references non-existent config key: service-ca.crt Aug 04 03:13:51.401800 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401783 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-certificates\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.402048 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f2069c-9b92-451c-ac80-087ecf892a3d-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.402048 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401850 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13504d71-ed32-4373-b189-966cf1a7f494-config\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.402048 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-stats-auth\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.402048 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.401919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.402048 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.402002 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 03:13:51.402048 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-serving-cert\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.402048 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.402040 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:51.902029283 +0000 UTC m=+255.023098602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : secret "router-metrics-certs-default" not found Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrvf\" (UniqueName: \"kubernetes.io/projected/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-kube-api-access-hlrvf\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-bound-sa-token\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402126 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25t6h\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-kube-api-access-25t6h\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-tmp\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402172 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e01db6e9-90bd-4912-b4a5-7b365f90f78d-ca-trust-extracted\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-trusted-ca\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-installation-pull-secrets\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxtx\" (UniqueName: \"kubernetes.io/projected/42c1c927-755a-4827-99ca-58cd1cb90a46-kube-api-access-cmxtx\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/13504d71-ed32-4373-b189-966cf1a7f494-kube-api-access-z5xfb\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.402306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402299 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13504d71-ed32-4373-b189-966cf1a7f494-serving-cert\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.402696 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402317 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-snapshots\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.402696 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-default-certificate\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.402696 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.402454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-service-ca-bundle\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.404206 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.404187 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-stats-auth\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.404518 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.404498 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-default-certificate\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.410332 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.410313 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxtx\" (UniqueName: \"kubernetes.io/projected/42c1c927-755a-4827-99ca-58cd1cb90a46-kube-api-access-cmxtx\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.503052 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.502972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrvf\" (UniqueName: \"kubernetes.io/projected/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-kube-api-access-hlrvf\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.503052 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-bound-sa-token\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.503052 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.503113 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.503124 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8965f5cc-t6xm7: secret "image-registry-tls" not found Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25t6h\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-kube-api-access-25t6h\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.503190 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls podName:e01db6e9-90bd-4912-b4a5-7b365f90f78d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:52.00316883 +0000 UTC m=+255.124238149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls") pod "image-registry-6f8965f5cc-t6xm7" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d") : secret "image-registry-tls" not found Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-tmp\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e01db6e9-90bd-4912-b4a5-7b365f90f78d-ca-trust-extracted\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-trusted-ca\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.503313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503290 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-installation-pull-secrets\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/13504d71-ed32-4373-b189-966cf1a7f494-kube-api-access-z5xfb\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13504d71-ed32-4373-b189-966cf1a7f494-serving-cert\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-snapshots\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-service-ca-bundle\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-trusted-ca-bundle\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-tmp\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e01db6e9-90bd-4912-b4a5-7b365f90f78d-ca-trust-extracted\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13504d71-ed32-4373-b189-966cf1a7f494-trusted-ca\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.503688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9f2069c-9b92-451c-ac80-087ecf892a3d-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdvgs\" (UniqueName: \"kubernetes.io/projected/b9f2069c-9b92-451c-ac80-087ecf892a3d-kube-api-access-xdvgs\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-image-registry-private-configuration\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-certificates\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503821 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f2069c-9b92-451c-ac80-087ecf892a3d-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13504d71-ed32-4373-b189-966cf1a7f494-config\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-serving-cert\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.503997 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-snapshots\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.504194 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.504157 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-service-ca-bundle\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.504571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.504281 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-trusted-ca-bundle\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.504571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.504294 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9f2069c-9b92-451c-ac80-087ecf892a3d-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.504571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.504353 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-trusted-ca\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.504571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.504475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-certificates\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.505067 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.505041 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13504d71-ed32-4373-b189-966cf1a7f494-config\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.505204 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.505181 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13504d71-ed32-4373-b189-966cf1a7f494-trusted-ca\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.506435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.506409 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-serving-cert\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.506535 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.506452 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f2069c-9b92-451c-ac80-087ecf892a3d-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.506604 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.506591 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-image-registry-private-configuration\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.506707 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.506687 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13504d71-ed32-4373-b189-966cf1a7f494-serving-cert\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.506844 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.506827 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-installation-pull-secrets\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.511721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.511695 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-bound-sa-token\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.511873 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.511854 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrvf\" (UniqueName: \"kubernetes.io/projected/b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4-kube-api-access-hlrvf\") pod \"insights-operator-77696685b-fvfjv\" (UID: \"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4\") " pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.512354 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.512327 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdvgs\" (UniqueName: \"kubernetes.io/projected/b9f2069c-9b92-451c-ac80-087ecf892a3d-kube-api-access-xdvgs\") pod \"kube-storage-version-migrator-operator-7878f4b88c-srg2h\" (UID: \"b9f2069c-9b92-451c-ac80-087ecf892a3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.512694 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.512679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5xfb\" (UniqueName: \"kubernetes.io/projected/13504d71-ed32-4373-b189-966cf1a7f494-kube-api-access-z5xfb\") pod \"console-operator-5ccd6b445f-hkqrp\" (UID: \"13504d71-ed32-4373-b189-966cf1a7f494\") " pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.512993 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.512979 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25t6h\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-kube-api-access-25t6h\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:51.573989 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.573955 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-fvfjv" Aug 04 03:13:51.582976 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.582957 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" Aug 04 03:13:51.589926 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.589886 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:13:51.712244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.712164 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-fvfjv"] Aug 04 03:13:51.715237 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:13:51.715217 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8aab2c4_5624_4d78_bfd0_1a31d19ab9b4.slice/crio-43f4be092a88a5066bf03a924bb333292ffcd06c73b5d887de8b409c9e4fc064 WatchSource:0}: Error finding container 43f4be092a88a5066bf03a924bb333292ffcd06c73b5d887de8b409c9e4fc064: Status 404 returned error can't find the container with id 43f4be092a88a5066bf03a924bb333292ffcd06c73b5d887de8b409c9e4fc064 Aug 04 03:13:51.907292 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.907263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.907502 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.907322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:51.907502 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.907419 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 03:13:51.907502 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.907448 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:52.907431117 +0000 UTC m=+256.028500437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : configmap references non-existent config key: service-ca.crt Aug 04 03:13:51.907502 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:51.907472 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:52.907464953 +0000 UTC m=+256.028534273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : secret "router-metrics-certs-default" not found Aug 04 03:13:51.932714 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.932688 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-hkqrp"] Aug 04 03:13:51.935417 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:13:51.935363 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13504d71_ed32_4373_b189_966cf1a7f494.slice/crio-f2859471edd6b82f9ff1e2d0cae0d3ea5ce2c9aceee923bf555d949d04167e7c WatchSource:0}: Error finding container f2859471edd6b82f9ff1e2d0cae0d3ea5ce2c9aceee923bf555d949d04167e7c: Status 404 returned error can't find the container with id f2859471edd6b82f9ff1e2d0cae0d3ea5ce2c9aceee923bf555d949d04167e7c Aug 04 03:13:51.936622 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:51.936598 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h"] Aug 04 03:13:51.939104 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:13:51.939079 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f2069c_9b92_451c_ac80_087ecf892a3d.slice/crio-753127dac8a2f5369ea2477c35109d2552ff9dfe79b41e827cdb1e86bda0c0f9 WatchSource:0}: Error finding container 753127dac8a2f5369ea2477c35109d2552ff9dfe79b41e827cdb1e86bda0c0f9: Status 404 returned error can't find the container with id 753127dac8a2f5369ea2477c35109d2552ff9dfe79b41e827cdb1e86bda0c0f9 Aug 04 03:13:52.007908 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:52.007864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:52.008064 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:52.008037 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 03:13:52.008064 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:52.008057 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8965f5cc-t6xm7: secret "image-registry-tls" not found Aug 04 03:13:52.008174 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:52.008120 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls podName:e01db6e9-90bd-4912-b4a5-7b365f90f78d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:53.008100343 +0000 UTC m=+256.129169683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls") pod "image-registry-6f8965f5cc-t6xm7" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d") : secret "image-registry-tls" not found Aug 04 03:13:52.186559 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:52.186474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fvfjv" event={"ID":"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4","Type":"ContainerStarted","Data":"43f4be092a88a5066bf03a924bb333292ffcd06c73b5d887de8b409c9e4fc064"} Aug 04 03:13:52.187579 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:52.187550 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" event={"ID":"13504d71-ed32-4373-b189-966cf1a7f494","Type":"ContainerStarted","Data":"f2859471edd6b82f9ff1e2d0cae0d3ea5ce2c9aceee923bf555d949d04167e7c"} Aug 04 03:13:52.188491 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:52.188473 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" event={"ID":"b9f2069c-9b92-451c-ac80-087ecf892a3d","Type":"ContainerStarted","Data":"753127dac8a2f5369ea2477c35109d2552ff9dfe79b41e827cdb1e86bda0c0f9"} Aug 04 03:13:52.916201 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:52.915852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:52.916201 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:52.915957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:52.916201 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:52.916084 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 03:13:52.916201 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:52.916104 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:54.916034468 +0000 UTC m=+258.037103791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : configmap references non-existent config key: service-ca.crt Aug 04 03:13:52.916201 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:52.916136 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:54.916124854 +0000 UTC m=+258.037194180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : secret "router-metrics-certs-default" not found Aug 04 03:13:53.016798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:53.016734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:53.017062 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:53.017033 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 03:13:53.017062 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:53.017059 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8965f5cc-t6xm7: secret "image-registry-tls" not found Aug 04 03:13:53.017329 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:53.017129 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls podName:e01db6e9-90bd-4912-b4a5-7b365f90f78d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:55.017107698 +0000 UTC m=+258.138177020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls") pod "image-registry-6f8965f5cc-t6xm7" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d") : secret "image-registry-tls" not found Aug 04 03:13:54.933182 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:54.933159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:54.933406 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:54.933221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:54.933406 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:54.933325 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 03:13:54.933406 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:54.933329 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:58.933309832 +0000 UTC m=+262.054379175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : configmap references non-existent config key: service-ca.crt Aug 04 03:13:54.933406 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:54.933370 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:13:58.933359889 +0000 UTC m=+262.054429213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : secret "router-metrics-certs-default" not found Aug 04 03:13:55.034132 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.034101 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:55.034316 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:55.034229 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 03:13:55.034316 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:55.034241 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8965f5cc-t6xm7: secret "image-registry-tls" not found Aug 04 03:13:55.034316 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:55.034278 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls podName:e01db6e9-90bd-4912-b4a5-7b365f90f78d nodeName:}" failed. No retries permitted until 2026-08-04 03:13:59.034266729 +0000 UTC m=+262.155336052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls") pod "image-registry-6f8965f5cc-t6xm7" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d") : secret "image-registry-tls" not found Aug 04 03:13:55.199247 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.199165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fvfjv" event={"ID":"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4","Type":"ContainerStarted","Data":"8997fa35a7c91cf483c0c9b292d45350a360b45eb1d4affcc22a9ff814c1bcbc"} Aug 04 03:13:55.200805 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.200785 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/0.log" Aug 04 03:13:55.200957 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.200821 2568 generic.go:356] "Generic (PLEG): container finished" podID="13504d71-ed32-4373-b189-966cf1a7f494" containerID="2a8123867f6b010f6ba7faa8a1ff4a45451c793250c1963af8efbd724bca4795" exitCode=255 Aug 04 03:13:55.200957 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.200926 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" event={"ID":"13504d71-ed32-4373-b189-966cf1a7f494","Type":"ContainerDied","Data":"2a8123867f6b010f6ba7faa8a1ff4a45451c793250c1963af8efbd724bca4795"} Aug 04 03:13:55.201107 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.201093 2568 scope.go:117] "RemoveContainer" containerID="2a8123867f6b010f6ba7faa8a1ff4a45451c793250c1963af8efbd724bca4795" Aug 04 03:13:55.202518 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.202499 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" event={"ID":"b9f2069c-9b92-451c-ac80-087ecf892a3d","Type":"ContainerStarted","Data":"436c33e5544ab91cbdd6dbf58d65d82edc60de8d76771a47a41b6c18d17bade0"} Aug 04 03:13:55.215792 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.215483 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-fvfjv" podStartSLOduration=0.98503305 podStartE2EDuration="4.215467919s" totalImagesPullingTime="3.230434869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:13:55.215020602 +0000 UTC m=+258.336089960" watchObservedRunningTime="2026-08-04 03:13:55.215467919 +0000 UTC m=+258.336537262" Aug 04 03:13:55.232076 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.232031 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" podStartSLOduration=1.222132947 podStartE2EDuration="4.23201234s" totalImagesPullingTime="3.009879393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:13:55.230741728 +0000 UTC m=+258.351811070" watchObservedRunningTime="2026-08-04 03:13:55.23201234 +0000 UTC m=+258.353081683" Aug 04 03:13:55.980037 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.980002 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lxb82"] Aug 04 03:13:55.983524 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.983507 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:55.985849 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.985826 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2m679\"" Aug 04 03:13:55.985849 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.985835 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 03:13:55.986036 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.985950 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 03:13:55.991303 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:55.991281 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lxb82"] Aug 04 03:13:56.144055 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.144021 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/870d694c-7f00-4f97-b9a4-258f94e1c31e-data-volume\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.144227 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.144121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/870d694c-7f00-4f97-b9a4-258f94e1c31e-crio-socket\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.144227 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.144174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.144227 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.144203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.144227 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.144219 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rj4\" (UniqueName: \"kubernetes.io/projected/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-api-access-v8rj4\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.209491 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.209462 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:13:56.209808 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.209795 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/0.log" Aug 04 03:13:56.209866 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.209825 2568 generic.go:356] "Generic (PLEG): container finished" podID="13504d71-ed32-4373-b189-966cf1a7f494" containerID="9e94577cc683854b6da876acbbf947ca28e9730914515c957e46db42a3d644b7" exitCode=255 Aug 04 03:13:56.209926 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.209855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" event={"ID":"13504d71-ed32-4373-b189-966cf1a7f494","Type":"ContainerDied","Data":"9e94577cc683854b6da876acbbf947ca28e9730914515c957e46db42a3d644b7"} Aug 04 03:13:56.209966 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.209935 2568 scope.go:117] "RemoveContainer" containerID="2a8123867f6b010f6ba7faa8a1ff4a45451c793250c1963af8efbd724bca4795" Aug 04 03:13:56.210199 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.210183 2568 scope.go:117] "RemoveContainer" containerID="9e94577cc683854b6da876acbbf947ca28e9730914515c957e46db42a3d644b7" Aug 04 03:13:56.210379 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.210361 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hkqrp_openshift-console-operator(13504d71-ed32-4373-b189-966cf1a7f494)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" podUID="13504d71-ed32-4373-b189-966cf1a7f494" Aug 04 03:13:56.245183 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.245127 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.245183 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.245154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rj4\" (UniqueName: \"kubernetes.io/projected/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-api-access-v8rj4\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.245310 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.245211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/870d694c-7f00-4f97-b9a4-258f94e1c31e-data-volume\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.245310 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.245241 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 03:13:56.245310 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.245243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/870d694c-7f00-4f97-b9a4-258f94e1c31e-crio-socket\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.245310 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.245292 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/870d694c-7f00-4f97-b9a4-258f94e1c31e-crio-socket\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.245512 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.245306 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:13:56.745285537 +0000 UTC m=+259.866354869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : configmap "kube-rbac-proxy" not found Aug 04 03:13:56.245512 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.245397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.245512 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.245506 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/870d694c-7f00-4f97-b9a4-258f94e1c31e-data-volume\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.245644 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.245517 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 03:13:56.245644 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.245565 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:13:56.745552621 +0000 UTC m=+259.866621941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : secret "insights-runtime-extractor-tls" not found Aug 04 03:13:56.256553 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.256530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rj4\" (UniqueName: \"kubernetes.io/projected/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-api-access-v8rj4\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.749804 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.749771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.750019 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:56.749819 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:56.750019 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.749949 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 03:13:56.750019 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.749974 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 03:13:56.750019 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.750012 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:13:57.749994699 +0000 UTC m=+260.871064019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : secret "insights-runtime-extractor-tls" not found Aug 04 03:13:56.750160 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:56.750027 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:13:57.75002099 +0000 UTC m=+260.871090309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : configmap "kube-rbac-proxy" not found Aug 04 03:13:57.213576 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:57.213547 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:13:57.213961 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:57.213837 2568 scope.go:117] "RemoveContainer" containerID="9e94577cc683854b6da876acbbf947ca28e9730914515c957e46db42a3d644b7" Aug 04 03:13:57.214012 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:57.213986 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hkqrp_openshift-console-operator(13504d71-ed32-4373-b189-966cf1a7f494)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" podUID="13504d71-ed32-4373-b189-966cf1a7f494" Aug 04 03:13:57.758371 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:57.758339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:57.758534 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:57.758379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:57.758534 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:57.758471 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 03:13:57.758534 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:57.758484 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 03:13:57.758534 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:57.758521 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:13:59.758508375 +0000 UTC m=+262.879577694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : configmap "kube-rbac-proxy" not found Aug 04 03:13:57.758667 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:57.758545 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:13:59.758532832 +0000 UTC m=+262.879602157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : secret "insights-runtime-extractor-tls" not found Aug 04 03:13:58.143853 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:58.143782 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vcc28_b2a765b1-015e-490f-8423-5779334c52a7/dns-node-resolver/0.log" Aug 04 03:13:58.943910 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:58.943824 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9vmv7_e2fb4df7-54b7-403d-bcbe-116f661d41c6/node-ca/0.log" Aug 04 03:13:58.968322 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:58.968294 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:58.968489 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:58.968471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:13:58.968545 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:58.968497 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:06.968475218 +0000 UTC m=+270.089544540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : configmap references non-existent config key: service-ca.crt Aug 04 03:13:58.968602 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:58.968551 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 03:13:58.968602 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:58.968601 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs podName:42c1c927-755a-4827-99ca-58cd1cb90a46 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:06.968587961 +0000 UTC m=+270.089657282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs") pod "router-default-885fb9fbb-7xtvr" (UID: "42c1c927-755a-4827-99ca-58cd1cb90a46") : secret "router-metrics-certs-default" not found Aug 04 03:13:59.069810 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:59.069773 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:13:59.070009 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:59.069936 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 03:13:59.070009 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:59.069956 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f8965f5cc-t6xm7: secret "image-registry-tls" not found Aug 04 03:13:59.070009 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:59.070006 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls podName:e01db6e9-90bd-4912-b4a5-7b365f90f78d nodeName:}" failed. No retries permitted until 2026-08-04 03:14:07.069991994 +0000 UTC m=+270.191061314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls") pod "image-registry-6f8965f5cc-t6xm7" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d") : secret "image-registry-tls" not found Aug 04 03:13:59.543143 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:59.543111 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mp8nz_4e8b9811-69f2-4b6a-ab51-1e200e549ff8/serve-healthcheck-canary/0.log" Aug 04 03:13:59.775791 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:59.775757 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:59.776008 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:13:59.775868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:13:59.776008 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:59.775888 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 03:13:59.776008 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:59.775982 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 03:13:59.776008 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:59.775994 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:14:03.775973106 +0000 UTC m=+266.897042441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : configmap "kube-rbac-proxy" not found Aug 04 03:13:59.776230 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:13:59.776036 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:14:03.776020603 +0000 UTC m=+266.897089923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : secret "insights-runtime-extractor-tls" not found Aug 04 03:14:01.590365 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:01.590277 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:14:01.590365 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:01.590336 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:14:01.590768 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:01.590660 2568 scope.go:117] "RemoveContainer" containerID="9e94577cc683854b6da876acbbf947ca28e9730914515c957e46db42a3d644b7" Aug 04 03:14:01.590818 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:01.590799 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-hkqrp_openshift-console-operator(13504d71-ed32-4373-b189-966cf1a7f494)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" podUID="13504d71-ed32-4373-b189-966cf1a7f494" Aug 04 03:14:03.812798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:03.812745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:14:03.812798 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:03.812804 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:14:03.813278 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:03.812872 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 03:14:03.813278 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:03.812937 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:14:11.812923162 +0000 UTC m=+274.933992486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : configmap "kube-rbac-proxy" not found Aug 04 03:14:03.813278 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:03.812933 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 03:14:03.813278 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:03.813020 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls podName:870d694c-7f00-4f97-b9a4-258f94e1c31e nodeName:}" failed. No retries permitted until 2026-08-04 03:14:11.813002326 +0000 UTC m=+274.934071647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lxb82" (UID: "870d694c-7f00-4f97-b9a4-258f94e1c31e") : secret "insights-runtime-extractor-tls" not found Aug 04 03:14:07.039046 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.039012 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:07.039433 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.039078 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:07.039631 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.039614 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1c927-755a-4827-99ca-58cd1cb90a46-service-ca-bundle\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:07.041411 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.041387 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c1c927-755a-4827-99ca-58cd1cb90a46-metrics-certs\") pod \"router-default-885fb9fbb-7xtvr\" (UID: \"42c1c927-755a-4827-99ca-58cd1cb90a46\") " pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:07.076799 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.076760 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:07.140556 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.140483 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:14:07.142860 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.142816 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"image-registry-6f8965f5cc-t6xm7\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:14:07.195701 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.195669 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:14:07.199692 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.199571 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-885fb9fbb-7xtvr"] Aug 04 03:14:07.203833 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:07.203808 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c1c927_755a_4827_99ca_58cd1cb90a46.slice/crio-770bd9124f41fb07150e9892c62a7f747e71ce7f79d203ca3cfa3c3c66e76595 WatchSource:0}: Error finding container 770bd9124f41fb07150e9892c62a7f747e71ce7f79d203ca3cfa3c3c66e76595: Status 404 returned error can't find the container with id 770bd9124f41fb07150e9892c62a7f747e71ce7f79d203ca3cfa3c3c66e76595 Aug 04 03:14:07.242375 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.242337 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" event={"ID":"42c1c927-755a-4827-99ca-58cd1cb90a46","Type":"ContainerStarted","Data":"770bd9124f41fb07150e9892c62a7f747e71ce7f79d203ca3cfa3c3c66e76595"} Aug 04 03:14:07.328641 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:07.328556 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8965f5cc-t6xm7"] Aug 04 03:14:07.331641 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:07.331612 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01db6e9_90bd_4912_b4a5_7b365f90f78d.slice/crio-e2e8b9a0f39b7e9efadec9ac9d79d130335c01f590040a70c70ddbd366102a91 WatchSource:0}: Error finding container e2e8b9a0f39b7e9efadec9ac9d79d130335c01f590040a70c70ddbd366102a91: Status 404 returned error can't find the container with id e2e8b9a0f39b7e9efadec9ac9d79d130335c01f590040a70c70ddbd366102a91 Aug 04 03:14:08.246650 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:08.246605 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" event={"ID":"42c1c927-755a-4827-99ca-58cd1cb90a46","Type":"ContainerStarted","Data":"7bccde1beacd01f992f9aedf9f3680f54c4744c9d67a8ec6a256459b663bf97b"} Aug 04 03:14:08.248098 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:08.248077 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" event={"ID":"e01db6e9-90bd-4912-b4a5-7b365f90f78d","Type":"ContainerStarted","Data":"604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7"} Aug 04 03:14:08.248190 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:08.248112 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" event={"ID":"e01db6e9-90bd-4912-b4a5-7b365f90f78d","Type":"ContainerStarted","Data":"e2e8b9a0f39b7e9efadec9ac9d79d130335c01f590040a70c70ddbd366102a91"} Aug 04 03:14:08.248242 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:08.248222 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:14:08.267275 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:08.267231 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" podStartSLOduration=17.267217535 podStartE2EDuration="17.267217535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:08.266178814 +0000 UTC m=+271.387248156" watchObservedRunningTime="2026-08-04 03:14:08.267217535 +0000 UTC m=+271.388286879" Aug 04 03:14:08.281956 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:08.281882 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" podStartSLOduration=17.281868255 podStartE2EDuration="17.281868255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:08.281542951 +0000 UTC m=+271.402612294" watchObservedRunningTime="2026-08-04 03:14:08.281868255 +0000 UTC m=+271.402937597" Aug 04 03:14:09.077757 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:09.077717 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:09.080280 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:09.080255 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:09.251306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:09.251274 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:09.253595 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:09.253575 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-885fb9fbb-7xtvr" Aug 04 03:14:11.877879 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:11.877841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:14:11.877879 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:11.877880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:14:11.878354 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:11.878338 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/870d694c-7f00-4f97-b9a4-258f94e1c31e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:14:11.880097 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:11.880074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/870d694c-7f00-4f97-b9a4-258f94e1c31e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lxb82\" (UID: \"870d694c-7f00-4f97-b9a4-258f94e1c31e\") " pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:14:11.893670 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:11.893649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lxb82" Aug 04 03:14:12.021587 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:12.021556 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lxb82"] Aug 04 03:14:12.025551 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:12.025526 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870d694c_7f00_4f97_b9a4_258f94e1c31e.slice/crio-bb733aaa43037cd91306674a20567633b5b2f7eaccfc48fdf18313a123a32714 WatchSource:0}: Error finding container bb733aaa43037cd91306674a20567633b5b2f7eaccfc48fdf18313a123a32714: Status 404 returned error can't find the container with id bb733aaa43037cd91306674a20567633b5b2f7eaccfc48fdf18313a123a32714 Aug 04 03:14:12.262075 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:12.262042 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxb82" event={"ID":"870d694c-7f00-4f97-b9a4-258f94e1c31e","Type":"ContainerStarted","Data":"98e5586a33e5ccbe5b1a8b26ecb92fdc113009a3c4adcc641aa672e643070f83"} Aug 04 03:14:12.262211 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:12.262096 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxb82" event={"ID":"870d694c-7f00-4f97-b9a4-258f94e1c31e","Type":"ContainerStarted","Data":"bb733aaa43037cd91306674a20567633b5b2f7eaccfc48fdf18313a123a32714"} Aug 04 03:14:13.270079 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:13.270039 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxb82" event={"ID":"870d694c-7f00-4f97-b9a4-258f94e1c31e","Type":"ContainerStarted","Data":"4ad80927fe05239b03c1e1c8ae9d69821e9f76ac97a09906e5e7cb2112729f6b"} Aug 04 03:14:14.275041 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:14.274940 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lxb82" event={"ID":"870d694c-7f00-4f97-b9a4-258f94e1c31e","Type":"ContainerStarted","Data":"f4a313a58ac790e5877ed5b2700f3cfa516143210d96e6e3d0a9a3590c3e1dc2"} Aug 04 03:14:14.292074 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:14.292032 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lxb82" podStartSLOduration=17.461602069 podStartE2EDuration="19.292017121s" totalImagesPullingTime="1.830415052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 03:13:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:14.29079138 +0000 UTC m=+277.411860721" watchObservedRunningTime="2026-08-04 03:14:14.292017121 +0000 UTC m=+277.413086462" Aug 04 03:14:14.367463 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:14.367433 2568 scope.go:117] "RemoveContainer" containerID="9e94577cc683854b6da876acbbf947ca28e9730914515c957e46db42a3d644b7" Aug 04 03:14:15.284573 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:15.284544 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:14:15.284977 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:15.284661 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" event={"ID":"13504d71-ed32-4373-b189-966cf1a7f494","Type":"ContainerStarted","Data":"0cf0b7c7a52af7057a79a34eb10938e72f5d11f4b44bc63a7f3a84bab34156ae"} Aug 04 03:14:15.285078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:15.285055 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:14:15.289652 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:15.289633 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" Aug 04 03:14:15.304085 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:15.304047 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-hkqrp" podStartSLOduration=21.28863831 podStartE2EDuration="24.304036123s" totalImagesPullingTime="3.015397813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:15.302989548 +0000 UTC m=+278.424058880" watchObservedRunningTime="2026-08-04 03:14:15.304036123 +0000 UTC m=+278.425105464" Aug 04 03:14:18.960605 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:18.960562 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-tsqgz" podUID="cbe2b01b-b6ca-46f2-a852-c087b167d470" Aug 04 03:14:19.298783 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:19.298696 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tsqgz" Aug 04 03:14:19.739180 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:19.739151 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8965f5cc-t6xm7"] Aug 04 03:14:19.757808 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:19.757722 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-6t57s"] Aug 04 03:14:19.761099 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:19.761081 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-6t57s" Aug 04 03:14:19.764723 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:19.764701 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-28.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-130-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Aug 04 03:14:19.764830 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:19.764798 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-130-28.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-130-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Aug 04 03:14:19.765493 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:19.765469 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-7v8z5\" is forbidden: User \"system:node:ip-10-0-130-28.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-130-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"default-dockercfg-7v8z5\"" type="*v1.Secret" Aug 04 03:14:19.789765 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:19.789742 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-6t57s"] Aug 04 03:14:19.839961 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:19.839928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xtr\" (UniqueName: \"kubernetes.io/projected/3d38fb25-500a-4f20-884f-16cb59d68818-kube-api-access-m7xtr\") pod \"downloads-77c974ffcf-6t57s\" (UID: \"3d38fb25-500a-4f20-884f-16cb59d68818\") " pod="openshift-console/downloads-77c974ffcf-6t57s" Aug 04 03:14:19.941196 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:19.941107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xtr\" (UniqueName: \"kubernetes.io/projected/3d38fb25-500a-4f20-884f-16cb59d68818-kube-api-access-m7xtr\") pod \"downloads-77c974ffcf-6t57s\" (UID: \"3d38fb25-500a-4f20-884f-16cb59d68818\") " pod="openshift-console/downloads-77c974ffcf-6t57s" Aug 04 03:14:20.779690 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:20.779660 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 03:14:21.108333 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:21.108255 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 03:14:21.120791 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:21.120760 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xtr\" (UniqueName: \"kubernetes.io/projected/3d38fb25-500a-4f20-884f-16cb59d68818-kube-api-access-m7xtr\") pod \"downloads-77c974ffcf-6t57s\" (UID: \"3d38fb25-500a-4f20-884f-16cb59d68818\") " pod="openshift-console/downloads-77c974ffcf-6t57s" Aug 04 03:14:21.328838 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:21.328806 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7v8z5\"" Aug 04 03:14:21.329802 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:21.329788 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-6t57s" Aug 04 03:14:21.454284 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:21.454241 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-6t57s"] Aug 04 03:14:21.456839 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:21.456815 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d38fb25_500a_4f20_884f_16cb59d68818.slice/crio-6170a5da3f45d3ac65836773bc2f003a6cadd3d28d0b19d5d4b813268f425595 WatchSource:0}: Error finding container 6170a5da3f45d3ac65836773bc2f003a6cadd3d28d0b19d5d4b813268f425595: Status 404 returned error can't find the container with id 6170a5da3f45d3ac65836773bc2f003a6cadd3d28d0b19d5d4b813268f425595 Aug 04 03:14:22.039041 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.039001 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-8m25v"] Aug 04 03:14:22.042363 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.042341 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.044843 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.044815 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 04 03:14:22.046096 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.046067 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 03:14:22.046226 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.046094 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 03:14:22.046226 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.046104 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 03:14:22.046226 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.046125 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qznns\"" Aug 04 03:14:22.046470 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.046455 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 04 03:14:22.050932 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.050853 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-8m25v"] Aug 04 03:14:22.157044 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.157008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/996b9c63-9d9b-49e5-884c-2328d201d306-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.157297 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.157066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57s4n\" (UniqueName: \"kubernetes.io/projected/996b9c63-9d9b-49e5-884c-2328d201d306-kube-api-access-57s4n\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.157297 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.157092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/996b9c63-9d9b-49e5-884c-2328d201d306-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.157297 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.157190 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/996b9c63-9d9b-49e5-884c-2328d201d306-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.258533 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.258493 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/996b9c63-9d9b-49e5-884c-2328d201d306-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.258723 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.258571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/996b9c63-9d9b-49e5-884c-2328d201d306-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.258723 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.258602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57s4n\" (UniqueName: \"kubernetes.io/projected/996b9c63-9d9b-49e5-884c-2328d201d306-kube-api-access-57s4n\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.258723 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.258626 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/996b9c63-9d9b-49e5-884c-2328d201d306-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.259339 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.259312 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/996b9c63-9d9b-49e5-884c-2328d201d306-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.261343 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.261320 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/996b9c63-9d9b-49e5-884c-2328d201d306-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.261470 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.261356 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/996b9c63-9d9b-49e5-884c-2328d201d306-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.266824 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.266792 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57s4n\" (UniqueName: \"kubernetes.io/projected/996b9c63-9d9b-49e5-884c-2328d201d306-kube-api-access-57s4n\") pod \"prometheus-operator-758d7cb874-8m25v\" (UID: \"996b9c63-9d9b-49e5-884c-2328d201d306\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.315272 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.315189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-6t57s" event={"ID":"3d38fb25-500a-4f20-884f-16cb59d68818","Type":"ContainerStarted","Data":"6170a5da3f45d3ac65836773bc2f003a6cadd3d28d0b19d5d4b813268f425595"} Aug 04 03:14:22.353998 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.353969 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" Aug 04 03:14:22.493752 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.493709 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-8m25v"] Aug 04 03:14:22.498069 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:22.498022 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996b9c63_9d9b_49e5_884c_2328d201d306.slice/crio-69e158157c59dbaa1ee9751f31e15301540297c8af3ace0d4d3391f43b45c349 WatchSource:0}: Error finding container 69e158157c59dbaa1ee9751f31e15301540297c8af3ace0d4d3391f43b45c349: Status 404 returned error can't find the container with id 69e158157c59dbaa1ee9751f31e15301540297c8af3ace0d4d3391f43b45c349 Aug 04 03:14:22.764036 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.764003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:14:22.766564 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.766542 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cbe2b01b-b6ca-46f2-a852-c087b167d470-metrics-tls\") pod \"dns-default-tsqgz\" (UID: \"cbe2b01b-b6ca-46f2-a852-c087b167d470\") " pod="openshift-dns/dns-default-tsqgz" Aug 04 03:14:22.902600 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.902567 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4d7vw\"" Aug 04 03:14:22.910858 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:22.910827 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tsqgz" Aug 04 03:14:23.056585 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:23.056500 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tsqgz"] Aug 04 03:14:23.059540 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:23.059513 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe2b01b_b6ca_46f2_a852_c087b167d470.slice/crio-2fa3fb05a579a5e205e8844a541f173e20260bf41a8d537efe2646ebfb19c023 WatchSource:0}: Error finding container 2fa3fb05a579a5e205e8844a541f173e20260bf41a8d537efe2646ebfb19c023: Status 404 returned error can't find the container with id 2fa3fb05a579a5e205e8844a541f173e20260bf41a8d537efe2646ebfb19c023 Aug 04 03:14:23.320263 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:23.320152 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" event={"ID":"996b9c63-9d9b-49e5-884c-2328d201d306","Type":"ContainerStarted","Data":"69e158157c59dbaa1ee9751f31e15301540297c8af3ace0d4d3391f43b45c349"} Aug 04 03:14:23.321712 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:23.321674 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tsqgz" event={"ID":"cbe2b01b-b6ca-46f2-a852-c087b167d470","Type":"ContainerStarted","Data":"2fa3fb05a579a5e205e8844a541f173e20260bf41a8d537efe2646ebfb19c023"} Aug 04 03:14:24.328162 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:24.328122 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" event={"ID":"996b9c63-9d9b-49e5-884c-2328d201d306","Type":"ContainerStarted","Data":"8f83b356ef35adc6ccf65b020a3abf1d053b387bf09dc63148291a0c4b34f8ed"} Aug 04 03:14:25.334229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:25.334141 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" event={"ID":"996b9c63-9d9b-49e5-884c-2328d201d306","Type":"ContainerStarted","Data":"23d705ec5091972038caf56d3163a6e9aa8943d6655cacf7042d7379eac03cde"} Aug 04 03:14:25.336095 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:25.336071 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tsqgz" event={"ID":"cbe2b01b-b6ca-46f2-a852-c087b167d470","Type":"ContainerStarted","Data":"1f5a900e3399fa8e356ec77dde785ad2f3aeccc16276d9aa2a4c3a53c1ad6033"} Aug 04 03:14:25.336229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:25.336120 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tsqgz" event={"ID":"cbe2b01b-b6ca-46f2-a852-c087b167d470","Type":"ContainerStarted","Data":"9317a7cf26e4c7732d7f3c209c337a28d2910ca04e24e9775b454a88f943bc64"} Aug 04 03:14:25.336229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:25.336150 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tsqgz" Aug 04 03:14:25.351529 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:25.351484 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-8m25v" podStartSLOduration=1.707528057 podStartE2EDuration="3.351468947s" totalImagesPullingTime="1.64394089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:25.349683961 +0000 UTC m=+288.470753304" watchObservedRunningTime="2026-08-04 03:14:25.351468947 +0000 UTC m=+288.472538289" Aug 04 03:14:25.365213 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:25.365167 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tsqgz" podStartSLOduration=251.625599515 podStartE2EDuration="4m13.365153264s" totalImagesPullingTime="1.739553749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:25.364511682 +0000 UTC m=+288.485581042" watchObservedRunningTime="2026-08-04 03:14:25.365153264 +0000 UTC m=+288.486222608" Aug 04 03:14:27.362949 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.362916 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k"] Aug 04 03:14:27.366529 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.366510 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.369188 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.369168 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 03:14:27.369304 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.369168 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 03:14:27.369647 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.369618 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-48vgc\"" Aug 04 03:14:27.379237 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.379203 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xpfhs"] Aug 04 03:14:27.382571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.382549 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k"] Aug 04 03:14:27.382679 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.382648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.386305 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.386283 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 03:14:27.386526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.386511 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 03:14:27.386742 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.386719 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h8mnb\"" Aug 04 03:14:27.386846 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.386790 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 03:14:27.504672 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504635 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/616ab198-16b9-4193-b6b2-522178036c5d-metrics-client-ca\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.504672 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjk9\" (UniqueName: \"kubernetes.io/projected/202ea3e0-881f-45b2-9a04-bdd8d46f58be-kube-api-access-hmjk9\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.504964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504712 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/202ea3e0-881f-45b2-9a04-bdd8d46f58be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.504964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/202ea3e0-881f-45b2-9a04-bdd8d46f58be-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.504964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-root\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.504964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-tls\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.504964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504821 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j576c\" (UniqueName: \"kubernetes.io/projected/616ab198-16b9-4193-b6b2-522178036c5d-kube-api-access-j576c\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.504964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-textfile\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.504964 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.505313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.504986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.505313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.505030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-wtmp\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.505313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.505057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/202ea3e0-881f-45b2-9a04-bdd8d46f58be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.505313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.505072 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-sys\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.605951 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.605917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/202ea3e0-881f-45b2-9a04-bdd8d46f58be-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.606111 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.605965 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-root\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606111 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.605990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-tls\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606111 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606038 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j576c\" (UniqueName: \"kubernetes.io/projected/616ab198-16b9-4193-b6b2-522178036c5d-kube-api-access-j576c\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606111 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-textfile\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606321 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606321 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:27.606156 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 03:14:27.606321 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606321 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606179 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-root\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606321 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:27.606228 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-tls podName:616ab198-16b9-4193-b6b2-522178036c5d nodeName:}" failed. No retries permitted until 2026-08-04 03:14:28.106204599 +0000 UTC m=+291.227273930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-tls") pod "node-exporter-xpfhs" (UID: "616ab198-16b9-4193-b6b2-522178036c5d") : secret "node-exporter-tls" not found Aug 04 03:14:27.606568 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-wtmp\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606568 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606409 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/202ea3e0-881f-45b2-9a04-bdd8d46f58be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.606568 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606434 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-sys\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606568 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606441 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-textfile\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606568 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/616ab198-16b9-4193-b6b2-522178036c5d-metrics-client-ca\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606568 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjk9\" (UniqueName: \"kubernetes.io/projected/202ea3e0-881f-45b2-9a04-bdd8d46f58be-kube-api-access-hmjk9\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.606855 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606569 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-wtmp\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606855 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/202ea3e0-881f-45b2-9a04-bdd8d46f58be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.606855 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/616ab198-16b9-4193-b6b2-522178036c5d-sys\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.606855 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.606687 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/202ea3e0-881f-45b2-9a04-bdd8d46f58be-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.607145 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.607098 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.607201 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.607139 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/616ab198-16b9-4193-b6b2-522178036c5d-metrics-client-ca\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.609363 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.609323 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/202ea3e0-881f-45b2-9a04-bdd8d46f58be-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.609468 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.609450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.609675 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.609652 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/202ea3e0-881f-45b2-9a04-bdd8d46f58be-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.614360 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.614305 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j576c\" (UniqueName: \"kubernetes.io/projected/616ab198-16b9-4193-b6b2-522178036c5d-kube-api-access-j576c\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:27.614467 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.614446 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjk9\" (UniqueName: \"kubernetes.io/projected/202ea3e0-881f-45b2-9a04-bdd8d46f58be-kube-api-access-hmjk9\") pod \"openshift-state-metrics-6bb4d7b4dd-z786k\" (UID: \"202ea3e0-881f-45b2-9a04-bdd8d46f58be\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.677208 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.677175 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" Aug 04 03:14:27.816780 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:27.816428 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k"] Aug 04 03:14:27.818868 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:27.818842 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202ea3e0_881f_45b2_9a04_bdd8d46f58be.slice/crio-46022e3563e3a1b7f1add5c046bd984929284513f9e20ffed7559ab9b4b6b039 WatchSource:0}: Error finding container 46022e3563e3a1b7f1add5c046bd984929284513f9e20ffed7559ab9b4b6b039: Status 404 returned error can't find the container with id 46022e3563e3a1b7f1add5c046bd984929284513f9e20ffed7559ab9b4b6b039 Aug 04 03:14:28.112308 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:28.112256 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-tls\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:28.114553 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:28.114526 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/616ab198-16b9-4193-b6b2-522178036c5d-node-exporter-tls\") pod \"node-exporter-xpfhs\" (UID: \"616ab198-16b9-4193-b6b2-522178036c5d\") " pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:28.292529 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:28.292484 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xpfhs" Aug 04 03:14:28.312455 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:28.312413 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616ab198_16b9_4193_b6b2_522178036c5d.slice/crio-d7c266c90a3b494359d16f5e85878b1834037f9fd5091f0a6ebf3df7e6ff5abb WatchSource:0}: Error finding container d7c266c90a3b494359d16f5e85878b1834037f9fd5091f0a6ebf3df7e6ff5abb: Status 404 returned error can't find the container with id d7c266c90a3b494359d16f5e85878b1834037f9fd5091f0a6ebf3df7e6ff5abb Aug 04 03:14:28.349820 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:28.349785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpfhs" event={"ID":"616ab198-16b9-4193-b6b2-522178036c5d","Type":"ContainerStarted","Data":"d7c266c90a3b494359d16f5e85878b1834037f9fd5091f0a6ebf3df7e6ff5abb"} Aug 04 03:14:28.351992 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:28.351960 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" event={"ID":"202ea3e0-881f-45b2-9a04-bdd8d46f58be","Type":"ContainerStarted","Data":"66ebc1efc5ee5dbf16f45441c03a8d82f82da358b81f055536b0250b377635fe"} Aug 04 03:14:28.352112 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:28.352015 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" event={"ID":"202ea3e0-881f-45b2-9a04-bdd8d46f58be","Type":"ContainerStarted","Data":"9c97c5a76268fa76167c4b588de606387f91aa8c32ff40240ce031d52cc8d812"} Aug 04 03:14:28.352112 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:28.352041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" event={"ID":"202ea3e0-881f-45b2-9a04-bdd8d46f58be","Type":"ContainerStarted","Data":"46022e3563e3a1b7f1add5c046bd984929284513f9e20ffed7559ab9b4b6b039"} Aug 04 03:14:29.745306 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:29.745273 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:14:30.345867 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.345838 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55b44b789c-wnqsm"] Aug 04 03:14:30.356948 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.356926 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.360398 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.360375 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cf85w\"" Aug 04 03:14:30.360629 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.360611 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 04 03:14:30.360858 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.360813 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55b44b789c-wnqsm"] Aug 04 03:14:30.360858 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.360828 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 04 03:14:30.361367 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.361070 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 04 03:14:30.361367 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.361154 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fbhv60c8v4i3j\"" Aug 04 03:14:30.361367 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.361073 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 04 03:14:30.361367 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.361343 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 04 03:14:30.365331 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.365302 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" event={"ID":"202ea3e0-881f-45b2-9a04-bdd8d46f58be","Type":"ContainerStarted","Data":"a3253304605f17258c93cf59713bdc9c7fc8390703f24ea55c4d814fc8174766"} Aug 04 03:14:30.367454 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.367432 2568 generic.go:356] "Generic (PLEG): container finished" podID="616ab198-16b9-4193-b6b2-522178036c5d" containerID="9b320e96c5da2d6c5fbf76ba7ef9d944dfd9847f67514df8752113bfbbffe821" exitCode=0 Aug 04 03:14:30.367553 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.367513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpfhs" event={"ID":"616ab198-16b9-4193-b6b2-522178036c5d","Type":"ContainerDied","Data":"9b320e96c5da2d6c5fbf76ba7ef9d944dfd9847f67514df8752113bfbbffe821"} Aug 04 03:14:30.408317 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.408278 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-z786k" podStartSLOduration=1.33369751 podStartE2EDuration="3.408263054s" totalImagesPullingTime="2.074565544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:30.40782935 +0000 UTC m=+293.528898692" watchObservedRunningTime="2026-08-04 03:14:30.408263054 +0000 UTC m=+293.529332396" Aug 04 03:14:30.431842 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.431800 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-grpc-tls\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.431968 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.431928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c838d367-f2be-4506-9d42-7193aa324e7f-metrics-client-ca\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.432171 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.432026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.432171 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.432118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.432331 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.432298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-tls\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.432393 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.432357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.432473 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.432449 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.432525 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.432503 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlczj\" (UniqueName: \"kubernetes.io/projected/c838d367-f2be-4506-9d42-7193aa324e7f-kube-api-access-nlczj\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533097 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlczj\" (UniqueName: \"kubernetes.io/projected/c838d367-f2be-4506-9d42-7193aa324e7f-kube-api-access-nlczj\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533230 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533102 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-grpc-tls\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533230 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c838d367-f2be-4506-9d42-7193aa324e7f-metrics-client-ca\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533230 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533198 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533230 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533430 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533249 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-tls\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533430 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.533430 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.533288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.534237 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.534211 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c838d367-f2be-4506-9d42-7193aa324e7f-metrics-client-ca\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.536052 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.536006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.536435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.536410 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-tls\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.536664 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.536642 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.536808 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.536778 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.536880 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.536837 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.537982 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.537963 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c838d367-f2be-4506-9d42-7193aa324e7f-secret-grpc-tls\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.541435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.541410 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlczj\" (UniqueName: \"kubernetes.io/projected/c838d367-f2be-4506-9d42-7193aa324e7f-kube-api-access-nlczj\") pod \"thanos-querier-55b44b789c-wnqsm\" (UID: \"c838d367-f2be-4506-9d42-7193aa324e7f\") " pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.673629 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.673599 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:30.832775 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:30.832709 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55b44b789c-wnqsm"] Aug 04 03:14:30.836074 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:30.836044 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc838d367_f2be_4506_9d42_7193aa324e7f.slice/crio-071caae01670e3ae82d2a97128c3c92180c96af3df444b3c51b765e1386c5d9b WatchSource:0}: Error finding container 071caae01670e3ae82d2a97128c3c92180c96af3df444b3c51b765e1386c5d9b: Status 404 returned error can't find the container with id 071caae01670e3ae82d2a97128c3c92180c96af3df444b3c51b765e1386c5d9b Aug 04 03:14:31.372480 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.372442 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" event={"ID":"c838d367-f2be-4506-9d42-7193aa324e7f","Type":"ContainerStarted","Data":"071caae01670e3ae82d2a97128c3c92180c96af3df444b3c51b765e1386c5d9b"} Aug 04 03:14:31.374690 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.374658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpfhs" event={"ID":"616ab198-16b9-4193-b6b2-522178036c5d","Type":"ContainerStarted","Data":"5226b52459a6e289caeaca3e266da61608e13d8be47cdcbb4b74a52226b9cd1d"} Aug 04 03:14:31.374823 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.374709 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xpfhs" event={"ID":"616ab198-16b9-4193-b6b2-522178036c5d","Type":"ContainerStarted","Data":"34bac8e21df113d8828aa055bf78f4e143d4814743769f66d364240950320586"} Aug 04 03:14:31.393720 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.393675 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xpfhs" podStartSLOduration=2.548658197 podStartE2EDuration="4.393659937s" totalImagesPullingTime="1.728092398s" totalInitContainerRuntime="116.909342ms" isStatefulPod=true podCreationTimestamp="2026-08-04 03:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:31.392259324 +0000 UTC m=+294.513328692" watchObservedRunningTime="2026-08-04 03:14:31.393659937 +0000 UTC m=+294.514729279" Aug 04 03:14:31.665613 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.665578 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7cc6d447fc-nz69k"] Aug 04 03:14:31.685264 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.685233 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc6d447fc-nz69k"] Aug 04 03:14:31.685429 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.685364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.689394 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.689373 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 03:14:31.689644 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.689626 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Aug 04 03:14:31.690327 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.689822 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d0icbqalhtbf9\"" Aug 04 03:14:31.690327 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.689991 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Aug 04 03:14:31.690327 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.690114 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-75d7h\"" Aug 04 03:14:31.690327 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.690235 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Aug 04 03:14:31.744424 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.744392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/32ee0e56-7da3-4201-a0a1-d528f27dc836-audit-log\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.744582 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.744438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-client-ca-bundle\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.744582 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.744484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-secret-metrics-server-client-certs\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.744582 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.744554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ee0e56-7da3-4201-a0a1-d528f27dc836-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.744760 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.744591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/32ee0e56-7da3-4201-a0a1-d528f27dc836-metrics-server-audit-profiles\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.744760 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.744620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-secret-metrics-server-tls\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.744760 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.744643 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nnd\" (UniqueName: \"kubernetes.io/projected/32ee0e56-7da3-4201-a0a1-d528f27dc836-kube-api-access-c6nnd\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.845709 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.845676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/32ee0e56-7da3-4201-a0a1-d528f27dc836-audit-log\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.846278 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.845718 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-client-ca-bundle\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.846278 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.845823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-secret-metrics-server-client-certs\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.846278 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.845915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ee0e56-7da3-4201-a0a1-d528f27dc836-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.846278 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.845964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/32ee0e56-7da3-4201-a0a1-d528f27dc836-metrics-server-audit-profiles\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.846278 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.845993 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-secret-metrics-server-tls\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.846278 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.846018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nnd\" (UniqueName: \"kubernetes.io/projected/32ee0e56-7da3-4201-a0a1-d528f27dc836-kube-api-access-c6nnd\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.846278 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.846130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/32ee0e56-7da3-4201-a0a1-d528f27dc836-audit-log\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.847246 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.847198 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32ee0e56-7da3-4201-a0a1-d528f27dc836-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.847726 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.847676 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/32ee0e56-7da3-4201-a0a1-d528f27dc836-metrics-server-audit-profiles\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.849232 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.849192 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-client-ca-bundle\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.850007 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.849985 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-secret-metrics-server-client-certs\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.850361 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.850336 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/32ee0e56-7da3-4201-a0a1-d528f27dc836-secret-metrics-server-tls\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.855152 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.855130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nnd\" (UniqueName: \"kubernetes.io/projected/32ee0e56-7da3-4201-a0a1-d528f27dc836-kube-api-access-c6nnd\") pod \"metrics-server-7cc6d447fc-nz69k\" (UID: \"32ee0e56-7da3-4201-a0a1-d528f27dc836\") " pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:31.998493 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:31.998406 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:32.144699 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.144665 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx"] Aug 04 03:14:32.163638 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.163601 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx"] Aug 04 03:14:32.163806 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.163706 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:32.166449 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.166204 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 03:14:32.166449 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.166226 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jshvl\"" Aug 04 03:14:32.251119 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.251036 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mgh\" (UniqueName: \"kubernetes.io/projected/38897539-e1d5-4243-8589-1603f24557e9-kube-api-access-c5mgh\") pod \"monitoring-plugin-6dffbbd67-fddjx\" (UID: \"38897539-e1d5-4243-8589-1603f24557e9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:32.251267 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.251207 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38897539-e1d5-4243-8589-1603f24557e9-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-fddjx\" (UID: \"38897539-e1d5-4243-8589-1603f24557e9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:32.352553 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.352521 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mgh\" (UniqueName: \"kubernetes.io/projected/38897539-e1d5-4243-8589-1603f24557e9-kube-api-access-c5mgh\") pod \"monitoring-plugin-6dffbbd67-fddjx\" (UID: \"38897539-e1d5-4243-8589-1603f24557e9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:32.352736 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.352613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38897539-e1d5-4243-8589-1603f24557e9-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-fddjx\" (UID: \"38897539-e1d5-4243-8589-1603f24557e9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:32.352811 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:32.352749 2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Aug 04 03:14:32.352873 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:32.352844 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38897539-e1d5-4243-8589-1603f24557e9-monitoring-plugin-cert podName:38897539-e1d5-4243-8589-1603f24557e9 nodeName:}" failed. No retries permitted until 2026-08-04 03:14:32.852810044 +0000 UTC m=+295.973879369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/38897539-e1d5-4243-8589-1603f24557e9-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-fddjx" (UID: "38897539-e1d5-4243-8589-1603f24557e9") : secret "monitoring-plugin-cert" not found Aug 04 03:14:32.361311 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.361281 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mgh\" (UniqueName: \"kubernetes.io/projected/38897539-e1d5-4243-8589-1603f24557e9-kube-api-access-c5mgh\") pod \"monitoring-plugin-6dffbbd67-fddjx\" (UID: \"38897539-e1d5-4243-8589-1603f24557e9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:32.856969 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.856932 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38897539-e1d5-4243-8589-1603f24557e9-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-fddjx\" (UID: \"38897539-e1d5-4243-8589-1603f24557e9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:32.859490 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:32.859464 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/38897539-e1d5-4243-8589-1603f24557e9-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-fddjx\" (UID: \"38897539-e1d5-4243-8589-1603f24557e9\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:33.076655 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:33.076620 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:33.369040 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:33.368983 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lgtdn" podUID="ed8629cd-8988-40bb-a461-2155943874ca" Aug 04 03:14:35.342905 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:35.342863 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tsqgz" Aug 04 03:14:39.607570 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:39.607507 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:14:39.608097 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:39.608067 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:14:39.626287 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:39.626264 2568 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 03:14:39.756638 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:39.756460 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx"] Aug 04 03:14:39.759831 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:39.759777 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38897539_e1d5_4243_8589_1603f24557e9.slice/crio-6d22b107a1c22630b35d3a512d158ae9b1f5cbbfbb0f05a15b9546324d2e364d WatchSource:0}: Error finding container 6d22b107a1c22630b35d3a512d158ae9b1f5cbbfbb0f05a15b9546324d2e364d: Status 404 returned error can't find the container with id 6d22b107a1c22630b35d3a512d158ae9b1f5cbbfbb0f05a15b9546324d2e364d Aug 04 03:14:39.762326 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:39.762307 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 03:14:39.778585 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:39.778557 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7cc6d447fc-nz69k"] Aug 04 03:14:39.782463 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:14:39.782428 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ee0e56_7da3_4201_a0a1_d528f27dc836.slice/crio-ce11fa47d9290f26d2d21a9886581f7c01e6fb4b6c8ff0273a0f596bb9c8d5fa WatchSource:0}: Error finding container ce11fa47d9290f26d2d21a9886581f7c01e6fb4b6c8ff0273a0f596bb9c8d5fa: Status 404 returned error can't find the container with id ce11fa47d9290f26d2d21a9886581f7c01e6fb4b6c8ff0273a0f596bb9c8d5fa Aug 04 03:14:40.438762 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.438723 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" event={"ID":"38897539-e1d5-4243-8589-1603f24557e9","Type":"ContainerStarted","Data":"6d22b107a1c22630b35d3a512d158ae9b1f5cbbfbb0f05a15b9546324d2e364d"} Aug 04 03:14:40.446947 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.446912 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" event={"ID":"c838d367-f2be-4506-9d42-7193aa324e7f","Type":"ContainerStarted","Data":"74297d419d9332a6751962478e982d3fbe33953494d2ec550fc698d8363c5fec"} Aug 04 03:14:40.447108 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.447027 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" event={"ID":"c838d367-f2be-4506-9d42-7193aa324e7f","Type":"ContainerStarted","Data":"0608f799f2444752f4f8c800e89fe4fb2d3a0a1a17710eb929aac4dadeab5c24"} Aug 04 03:14:40.447108 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.447067 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" event={"ID":"c838d367-f2be-4506-9d42-7193aa324e7f","Type":"ContainerStarted","Data":"814565ccded037bd2ea8331e4345b62429fba63f3e5571c455c15ae1922b6fb5"} Aug 04 03:14:40.450779 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.450681 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-6t57s" event={"ID":"3d38fb25-500a-4f20-884f-16cb59d68818","Type":"ContainerStarted","Data":"af9d30dcde346113c3d3b50a7b30bf18c7631a9d8583e528b4c7f3b9af57d9f0"} Aug 04 03:14:40.450779 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.450742 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-6t57s" Aug 04 03:14:40.454123 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.454059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" event={"ID":"32ee0e56-7da3-4201-a0a1-d528f27dc836","Type":"ContainerStarted","Data":"ce11fa47d9290f26d2d21a9886581f7c01e6fb4b6c8ff0273a0f596bb9c8d5fa"} Aug 04 03:14:40.475400 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.475332 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-6t57s" Aug 04 03:14:40.504933 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:40.504299 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-6t57s" podStartSLOduration=3.3288826719999998 podStartE2EDuration="21.504278441s" totalImagesPullingTime="18.175395769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:40.474614078 +0000 UTC m=+303.595683442" watchObservedRunningTime="2026-08-04 03:14:40.504278441 +0000 UTC m=+303.625347784" Aug 04 03:14:41.464541 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:41.464498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" event={"ID":"c838d367-f2be-4506-9d42-7193aa324e7f","Type":"ContainerStarted","Data":"4a246bef947b080537a50910b2715e5a2fa85b440b7200eb1d016a7049d9ef1c"} Aug 04 03:14:41.465037 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:41.464592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" event={"ID":"c838d367-f2be-4506-9d42-7193aa324e7f","Type":"ContainerStarted","Data":"8719a4708de53a363793eb01237a40e66da68af3828a41cb2abe7c190e1b964e"} Aug 04 03:14:41.465037 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:41.464643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" event={"ID":"c838d367-f2be-4506-9d42-7193aa324e7f","Type":"ContainerStarted","Data":"bbc1416d06ff23aafa40b58a77d93d2f25a63594f2f992e45e72146424760544"} Aug 04 03:14:41.488193 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:41.488107 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" podStartSLOduration=1.903162676 podStartE2EDuration="11.488085281s" totalImagesPullingTime="9.584922605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:41.48650227 +0000 UTC m=+304.607571638" watchObservedRunningTime="2026-08-04 03:14:41.488085281 +0000 UTC m=+304.609154624" Aug 04 03:14:42.468587 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:42.468558 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:43.474780 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:43.474745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" event={"ID":"38897539-e1d5-4243-8589-1603f24557e9","Type":"ContainerStarted","Data":"f3aaebb7ef4f6b6de8af2e6b17a4f72f3381f0f4d7e29a077e2fd6159c42604b"} Aug 04 03:14:43.475884 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:43.475845 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:43.476586 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:43.476556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" event={"ID":"32ee0e56-7da3-4201-a0a1-d528f27dc836","Type":"ContainerStarted","Data":"e9cede6873898869528ae6705b269ba23ccc3dd19b0ed11deaf5ca2f6f9e76ef"} Aug 04 03:14:43.481317 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:43.481288 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" Aug 04 03:14:43.494049 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:43.494007 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-fddjx" podStartSLOduration=8.75155868 podStartE2EDuration="11.493993474s" totalImagesPullingTime="2.742434794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:43.492611889 +0000 UTC m=+306.613681259" watchObservedRunningTime="2026-08-04 03:14:43.493993474 +0000 UTC m=+306.615062815" Aug 04 03:14:43.509648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:43.509604 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" podStartSLOduration=9.785937748 podStartE2EDuration="12.509590094s" totalImagesPullingTime="2.723652346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:14:43.508197605 +0000 UTC m=+306.629266974" watchObservedRunningTime="2026-08-04 03:14:43.509590094 +0000 UTC m=+306.630659437" Aug 04 03:14:44.758066 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:44.758017 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" podUID="e01db6e9-90bd-4912-b4a5-7b365f90f78d" containerName="registry" containerID="cri-o://604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7" gracePeriod=30 Aug 04 03:14:45.061693 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.061669 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:14:45.088113 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088086 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e01db6e9-90bd-4912-b4a5-7b365f90f78d-ca-trust-extracted\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.088256 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088130 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.088256 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088156 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-bound-sa-token\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.088256 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088178 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25t6h\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-kube-api-access-25t6h\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.088256 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088213 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-certificates\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.088256 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088245 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-trusted-ca\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.088528 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088333 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-installation-pull-secrets\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.088528 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.088363 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-image-registry-private-configuration\") pod \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\" (UID: \"e01db6e9-90bd-4912-b4a5-7b365f90f78d\") " Aug 04 03:14:45.089712 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.089661 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 03:14:45.090095 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.090068 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 03:14:45.091619 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.091512 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:14:45.091979 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.091942 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:14:45.093861 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.093835 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-kube-api-access-25t6h" (OuterVolumeSpecName: "kube-api-access-25t6h") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "kube-api-access-25t6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:14:45.093982 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.093933 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 03:14:45.094206 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.094170 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 03:14:45.101162 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.101110 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01db6e9-90bd-4912-b4a5-7b365f90f78d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e01db6e9-90bd-4912-b4a5-7b365f90f78d" (UID: "e01db6e9-90bd-4912-b4a5-7b365f90f78d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:14:45.189981 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.189945 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-bound-sa-token\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.189981 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.189974 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25t6h\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-kube-api-access-25t6h\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.190202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.189989 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-certificates\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.190202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.190003 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e01db6e9-90bd-4912-b4a5-7b365f90f78d-trusted-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.190202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.190012 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-installation-pull-secrets\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.190202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.190020 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e01db6e9-90bd-4912-b4a5-7b365f90f78d-image-registry-private-configuration\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.190202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.190030 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e01db6e9-90bd-4912-b4a5-7b365f90f78d-ca-trust-extracted\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.190202 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.190041 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e01db6e9-90bd-4912-b4a5-7b365f90f78d-registry-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:14:45.486417 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.486376 2568 generic.go:356] "Generic (PLEG): container finished" podID="e01db6e9-90bd-4912-b4a5-7b365f90f78d" containerID="604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7" exitCode=0 Aug 04 03:14:45.486576 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.486445 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" Aug 04 03:14:45.486576 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.486474 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" event={"ID":"e01db6e9-90bd-4912-b4a5-7b365f90f78d","Type":"ContainerDied","Data":"604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7"} Aug 04 03:14:45.486576 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.486537 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8965f5cc-t6xm7" event={"ID":"e01db6e9-90bd-4912-b4a5-7b365f90f78d","Type":"ContainerDied","Data":"e2e8b9a0f39b7e9efadec9ac9d79d130335c01f590040a70c70ddbd366102a91"} Aug 04 03:14:45.486576 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.486561 2568 scope.go:117] "RemoveContainer" containerID="604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7" Aug 04 03:14:45.504679 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.504651 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f8965f5cc-t6xm7"] Aug 04 03:14:45.506748 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.506723 2568 scope.go:117] "RemoveContainer" containerID="604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7" Aug 04 03:14:45.507156 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:14:45.507131 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7\": container with ID starting with 604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7 not found: ID does not exist" containerID="604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7" Aug 04 03:14:45.507255 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.507165 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7"} err="failed to get container status \"604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7\": rpc error: code = NotFound desc = could not find container \"604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7\": container with ID starting with 604e2d739d1dcf76f97c45a2eeec2feca78c574a1deac12b24eddb71256cdce7 not found: ID does not exist" Aug 04 03:14:45.509834 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:45.509813 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f8965f5cc-t6xm7"] Aug 04 03:14:46.367620 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:46.367583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:14:47.374000 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:47.373966 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01db6e9-90bd-4912-b4a5-7b365f90f78d" path="/var/lib/kubelet/pods/e01db6e9-90bd-4912-b4a5-7b365f90f78d/volumes" Aug 04 03:14:48.483761 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:48.483734 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55b44b789c-wnqsm" Aug 04 03:14:51.998822 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:51.998784 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:14:51.999340 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:14:51.998852 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:15:05.561336 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:05.561301 2568 generic.go:356] "Generic (PLEG): container finished" podID="b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4" containerID="8997fa35a7c91cf483c0c9b292d45350a360b45eb1d4affcc22a9ff814c1bcbc" exitCode=0 Aug 04 03:15:05.561844 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:05.561382 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fvfjv" event={"ID":"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4","Type":"ContainerDied","Data":"8997fa35a7c91cf483c0c9b292d45350a360b45eb1d4affcc22a9ff814c1bcbc"} Aug 04 03:15:05.561844 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:05.561776 2568 scope.go:117] "RemoveContainer" containerID="8997fa35a7c91cf483c0c9b292d45350a360b45eb1d4affcc22a9ff814c1bcbc" Aug 04 03:15:06.567681 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:06.567645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-fvfjv" event={"ID":"b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4","Type":"ContainerStarted","Data":"8d2aba2a9234d5ba5f09a08763a64cacd43e3f53323ab5966852ed637b87733a"} Aug 04 03:15:12.004762 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:12.004732 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:15:12.008511 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:12.008492 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7cc6d447fc-nz69k" Aug 04 03:15:25.638041 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:25.638010 2568 generic.go:356] "Generic (PLEG): container finished" podID="b9f2069c-9b92-451c-ac80-087ecf892a3d" containerID="436c33e5544ab91cbdd6dbf58d65d82edc60de8d76771a47a41b6c18d17bade0" exitCode=0 Aug 04 03:15:25.638519 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:25.638082 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" event={"ID":"b9f2069c-9b92-451c-ac80-087ecf892a3d","Type":"ContainerDied","Data":"436c33e5544ab91cbdd6dbf58d65d82edc60de8d76771a47a41b6c18d17bade0"} Aug 04 03:15:25.638519 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:25.638461 2568 scope.go:117] "RemoveContainer" containerID="436c33e5544ab91cbdd6dbf58d65d82edc60de8d76771a47a41b6c18d17bade0" Aug 04 03:15:26.643339 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:26.643303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-srg2h" event={"ID":"b9f2069c-9b92-451c-ac80-087ecf892a3d","Type":"ContainerStarted","Data":"ea34fe2d0ee86749737ccecc9eaf2460af30922050f51e444e2c970910a47839"} Aug 04 03:15:51.230232 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:51.230190 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:15:51.232420 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:51.232402 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed8629cd-8988-40bb-a461-2155943874ca-metrics-certs\") pod \"network-metrics-daemon-lgtdn\" (UID: \"ed8629cd-8988-40bb-a461-2155943874ca\") " pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:15:51.470651 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:51.470622 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6l47c\"" Aug 04 03:15:51.478828 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:51.478806 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lgtdn" Aug 04 03:15:51.602699 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:51.602670 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lgtdn"] Aug 04 03:15:51.605942 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:15:51.605921 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8629cd_8988_40bb_a461_2155943874ca.slice/crio-8e617453dcd2ab5d7f2cf27cb639c31db474c1651ade284d3c0ff8cac879ba07 WatchSource:0}: Error finding container 8e617453dcd2ab5d7f2cf27cb639c31db474c1651ade284d3c0ff8cac879ba07: Status 404 returned error can't find the container with id 8e617453dcd2ab5d7f2cf27cb639c31db474c1651ade284d3c0ff8cac879ba07 Aug 04 03:15:51.738518 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:51.738377 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lgtdn" event={"ID":"ed8629cd-8988-40bb-a461-2155943874ca","Type":"ContainerStarted","Data":"8e617453dcd2ab5d7f2cf27cb639c31db474c1651ade284d3c0ff8cac879ba07"} Aug 04 03:15:52.743370 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:52.743343 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lgtdn" event={"ID":"ed8629cd-8988-40bb-a461-2155943874ca","Type":"ContainerStarted","Data":"75a514cb03e3b2ea1c6b17f8840d7f85bb95652c3bbc0a370d6185adfc55fbf0"} Aug 04 03:15:53.753132 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:53.753094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lgtdn" event={"ID":"ed8629cd-8988-40bb-a461-2155943874ca","Type":"ContainerStarted","Data":"f0e83001010bbc91368a32e02e6c036646aa719cfa112b2666b5bedb2e8ecd52"} Aug 04 03:15:53.768643 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:15:53.768599 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lgtdn" podStartSLOduration=375.788981406 podStartE2EDuration="6m16.768584844s" totalImagesPullingTime="979.603438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:15:53.768352704 +0000 UTC m=+376.889422063" watchObservedRunningTime="2026-08-04 03:15:53.768584844 +0000 UTC m=+376.889654180" Aug 04 03:17:00.914365 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.914281 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z"] Aug 04 03:17:00.914832 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.914601 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01db6e9-90bd-4912-b4a5-7b365f90f78d" containerName="registry" Aug 04 03:17:00.914832 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.914612 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01db6e9-90bd-4912-b4a5-7b365f90f78d" containerName="registry" Aug 04 03:17:00.914832 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.914660 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01db6e9-90bd-4912-b4a5-7b365f90f78d" containerName="registry" Aug 04 03:17:00.916850 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.916824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:00.919362 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.919345 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 03:17:00.920528 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.920506 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 03:17:00.920677 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.920523 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zztjk\"" Aug 04 03:17:00.926838 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.926816 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z"] Aug 04 03:17:00.969338 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.969311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:00.969508 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.969379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r792z\" (UniqueName: \"kubernetes.io/projected/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-kube-api-access-r792z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:00.969508 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:00.969405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.070012 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.069978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.070178 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.070036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r792z\" (UniqueName: \"kubernetes.io/projected/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-kube-api-access-r792z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.070178 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.070053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.070363 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.070343 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.070363 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.070366 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.078263 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.078242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r792z\" (UniqueName: \"kubernetes.io/projected/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-kube-api-access-r792z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.225643 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.225607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:01.350197 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:01.350165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z"] Aug 04 03:17:01.352795 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:17:01.352770 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f4b0b1_09e3_4e81_a0c1_2a557b8da784.slice/crio-694e4057ef413a1039f5582afbc4167ab7d45eb7aa98789bccf67ddb9d24a731 WatchSource:0}: Error finding container 694e4057ef413a1039f5582afbc4167ab7d45eb7aa98789bccf67ddb9d24a731: Status 404 returned error can't find the container with id 694e4057ef413a1039f5582afbc4167ab7d45eb7aa98789bccf67ddb9d24a731 Aug 04 03:17:02.003027 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:02.002986 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" event={"ID":"06f4b0b1-09e3-4e81-a0c1-2a557b8da784","Type":"ContainerStarted","Data":"694e4057ef413a1039f5582afbc4167ab7d45eb7aa98789bccf67ddb9d24a731"} Aug 04 03:17:08.029612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:08.029576 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" event={"ID":"06f4b0b1-09e3-4e81-a0c1-2a557b8da784","Type":"ContainerStarted","Data":"62ff527c5b747437b3ef0b684f48382f0548eafa1e99e992c28ca517a0135715"} Aug 04 03:17:09.034600 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:09.034563 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerID="62ff527c5b747437b3ef0b684f48382f0548eafa1e99e992c28ca517a0135715" exitCode=0 Aug 04 03:17:09.035104 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:09.034615 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" event={"ID":"06f4b0b1-09e3-4e81-a0c1-2a557b8da784","Type":"ContainerDied","Data":"62ff527c5b747437b3ef0b684f48382f0548eafa1e99e992c28ca517a0135715"} Aug 04 03:17:11.044687 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:11.044653 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerID="872775d71c40a39f08761731d530ad0be847706b9a8ae38bdafd394f2997a194" exitCode=0 Aug 04 03:17:11.045076 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:11.044707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" event={"ID":"06f4b0b1-09e3-4e81-a0c1-2a557b8da784","Type":"ContainerDied","Data":"872775d71c40a39f08761731d530ad0be847706b9a8ae38bdafd394f2997a194"} Aug 04 03:17:19.081341 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:19.081305 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerID="ea9978494b5dfd53736070d9b8d08c57da984cc0d61c0e2cf6791b9e589b922d" exitCode=0 Aug 04 03:17:19.081738 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:19.081388 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" event={"ID":"06f4b0b1-09e3-4e81-a0c1-2a557b8da784","Type":"ContainerDied","Data":"ea9978494b5dfd53736070d9b8d08c57da984cc0d61c0e2cf6791b9e589b922d"} Aug 04 03:17:20.208367 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.208345 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:20.337596 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.337504 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-bundle\") pod \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " Aug 04 03:17:20.337596 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.337538 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-util\") pod \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " Aug 04 03:17:20.337596 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.337560 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r792z\" (UniqueName: \"kubernetes.io/projected/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-kube-api-access-r792z\") pod \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\" (UID: \"06f4b0b1-09e3-4e81-a0c1-2a557b8da784\") " Aug 04 03:17:20.338367 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.338340 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-bundle" (OuterVolumeSpecName: "bundle") pod "06f4b0b1-09e3-4e81-a0c1-2a557b8da784" (UID: "06f4b0b1-09e3-4e81-a0c1-2a557b8da784"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:17:20.339726 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.339692 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-kube-api-access-r792z" (OuterVolumeSpecName: "kube-api-access-r792z") pod "06f4b0b1-09e3-4e81-a0c1-2a557b8da784" (UID: "06f4b0b1-09e3-4e81-a0c1-2a557b8da784"). InnerVolumeSpecName "kube-api-access-r792z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:17:20.344829 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.344799 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-util" (OuterVolumeSpecName: "util") pod "06f4b0b1-09e3-4e81-a0c1-2a557b8da784" (UID: "06f4b0b1-09e3-4e81-a0c1-2a557b8da784"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:17:20.438274 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.438240 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:20.438274 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.438269 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-util\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:20.438460 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:20.438282 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r792z\" (UniqueName: \"kubernetes.io/projected/06f4b0b1-09e3-4e81-a0c1-2a557b8da784-kube-api-access-r792z\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:21.090635 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:21.090607 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" Aug 04 03:17:21.090635 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:21.090611 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mjp9z" event={"ID":"06f4b0b1-09e3-4e81-a0c1-2a557b8da784","Type":"ContainerDied","Data":"694e4057ef413a1039f5582afbc4167ab7d45eb7aa98789bccf67ddb9d24a731"} Aug 04 03:17:21.090822 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:21.090654 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="694e4057ef413a1039f5582afbc4167ab7d45eb7aa98789bccf67ddb9d24a731" Aug 04 03:17:28.874336 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874301 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg"] Aug 04 03:17:28.874785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874603 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerName="extract" Aug 04 03:17:28.874785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874614 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerName="extract" Aug 04 03:17:28.874785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874629 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerName="pull" Aug 04 03:17:28.874785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874635 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerName="pull" Aug 04 03:17:28.874785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874643 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerName="util" Aug 04 03:17:28.874785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874649 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerName="util" Aug 04 03:17:28.874785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.874703 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f4b0b1-09e3-4e81-a0c1-2a557b8da784" containerName="extract" Aug 04 03:17:28.880375 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.880357 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:28.883046 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.883022 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 03:17:28.883046 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.883046 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zx5r7\"" Aug 04 03:17:28.883215 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.883071 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 03:17:28.889229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:28.889208 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg"] Aug 04 03:17:29.012809 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.012770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqnhg\" (UniqueName: \"kubernetes.io/projected/8cf9272e-b56c-477b-b0e7-c833d628d5fa-kube-api-access-zqnhg\") pod \"cert-manager-operator-controller-manager-55c444bf86-wzklg\" (UID: \"8cf9272e-b56c-477b-b0e7-c833d628d5fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:29.013031 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.012827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf9272e-b56c-477b-b0e7-c833d628d5fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wzklg\" (UID: \"8cf9272e-b56c-477b-b0e7-c833d628d5fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:29.114228 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.114183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf9272e-b56c-477b-b0e7-c833d628d5fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wzklg\" (UID: \"8cf9272e-b56c-477b-b0e7-c833d628d5fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:29.114367 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.114259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqnhg\" (UniqueName: \"kubernetes.io/projected/8cf9272e-b56c-477b-b0e7-c833d628d5fa-kube-api-access-zqnhg\") pod \"cert-manager-operator-controller-manager-55c444bf86-wzklg\" (UID: \"8cf9272e-b56c-477b-b0e7-c833d628d5fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:29.114614 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.114599 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cf9272e-b56c-477b-b0e7-c833d628d5fa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wzklg\" (UID: \"8cf9272e-b56c-477b-b0e7-c833d628d5fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:29.122854 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.122833 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqnhg\" (UniqueName: \"kubernetes.io/projected/8cf9272e-b56c-477b-b0e7-c833d628d5fa-kube-api-access-zqnhg\") pod \"cert-manager-operator-controller-manager-55c444bf86-wzklg\" (UID: \"8cf9272e-b56c-477b-b0e7-c833d628d5fa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:29.190009 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.189982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" Aug 04 03:17:29.331585 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:29.331558 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg"] Aug 04 03:17:29.333676 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:17:29.333647 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf9272e_b56c_477b_b0e7_c833d628d5fa.slice/crio-2233c444b5e47045005475a056d8b0b089b9bfc6f8c09b248b82d76f5c4aa3d0 WatchSource:0}: Error finding container 2233c444b5e47045005475a056d8b0b089b9bfc6f8c09b248b82d76f5c4aa3d0: Status 404 returned error can't find the container with id 2233c444b5e47045005475a056d8b0b089b9bfc6f8c09b248b82d76f5c4aa3d0 Aug 04 03:17:30.126180 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:30.126140 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" event={"ID":"8cf9272e-b56c-477b-b0e7-c833d628d5fa","Type":"ContainerStarted","Data":"2233c444b5e47045005475a056d8b0b089b9bfc6f8c09b248b82d76f5c4aa3d0"} Aug 04 03:17:31.132667 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:31.132633 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" event={"ID":"8cf9272e-b56c-477b-b0e7-c833d628d5fa","Type":"ContainerStarted","Data":"4ec6a731749bde6b8182a47fa089f05519a5c6b7e73a0463965793f059b12d8c"} Aug 04 03:17:31.156408 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:31.156345 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wzklg" podStartSLOduration=1.448803348 podStartE2EDuration="3.156323899s" totalImagesPullingTime="1.707520551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:17:31.155079356 +0000 UTC m=+474.276148695" watchObservedRunningTime="2026-08-04 03:17:31.156323899 +0000 UTC m=+474.277393242" Aug 04 03:17:32.477829 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.477793 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr"] Aug 04 03:17:32.480114 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.480092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.485565 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.485539 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 03:17:32.486698 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.486676 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zztjk\"" Aug 04 03:17:32.486832 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.486729 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 03:17:32.489051 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.489031 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr"] Aug 04 03:17:32.645559 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.645522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqpb\" (UniqueName: \"kubernetes.io/projected/27bfab85-ad29-4456-93f2-4d5058046514-kube-api-access-rdqpb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.645730 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.645568 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.645730 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.645599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.746928 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.746820 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.747083 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.746921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqpb\" (UniqueName: \"kubernetes.io/projected/27bfab85-ad29-4456-93f2-4d5058046514-kube-api-access-rdqpb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.747083 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.746970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.747234 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.747214 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.747271 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.747254 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.756643 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.756613 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqpb\" (UniqueName: \"kubernetes.io/projected/27bfab85-ad29-4456-93f2-4d5058046514-kube-api-access-rdqpb\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.795184 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.795150 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:32.920133 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:32.920109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr"] Aug 04 03:17:32.922333 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:17:32.922295 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bfab85_ad29_4456_93f2_4d5058046514.slice/crio-77248791ae7714a523cb812cbaef1ace6c9e39348f913a5131ddfa22f436fef0 WatchSource:0}: Error finding container 77248791ae7714a523cb812cbaef1ace6c9e39348f913a5131ddfa22f436fef0: Status 404 returned error can't find the container with id 77248791ae7714a523cb812cbaef1ace6c9e39348f913a5131ddfa22f436fef0 Aug 04 03:17:33.142694 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:33.142656 2568 generic.go:356] "Generic (PLEG): container finished" podID="27bfab85-ad29-4456-93f2-4d5058046514" containerID="dd8fecd2ebac853761427f88dea0de0681d5942173f62d522a1dfea4d219c3ce" exitCode=0 Aug 04 03:17:33.142859 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:33.142746 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" event={"ID":"27bfab85-ad29-4456-93f2-4d5058046514","Type":"ContainerDied","Data":"dd8fecd2ebac853761427f88dea0de0681d5942173f62d522a1dfea4d219c3ce"} Aug 04 03:17:33.142859 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:33.142800 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" event={"ID":"27bfab85-ad29-4456-93f2-4d5058046514","Type":"ContainerStarted","Data":"77248791ae7714a523cb812cbaef1ace6c9e39348f913a5131ddfa22f436fef0"} Aug 04 03:17:34.924210 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:34.924180 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7x24b"] Aug 04 03:17:34.930741 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:34.930709 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:34.933472 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:34.933432 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 03:17:34.933472 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:34.933451 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 03:17:34.934370 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:34.934353 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-h82t7\"" Aug 04 03:17:34.940207 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:34.940187 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7x24b"] Aug 04 03:17:35.065675 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.065618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprqm\" (UniqueName: \"kubernetes.io/projected/fd566afa-c0e6-48c3-894e-a515f08123b9-kube-api-access-qprqm\") pod \"cert-manager-cainjector-559cdd8c87-7x24b\" (UID: \"fd566afa-c0e6-48c3-894e-a515f08123b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:35.065675 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.065680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd566afa-c0e6-48c3-894e-a515f08123b9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7x24b\" (UID: \"fd566afa-c0e6-48c3-894e-a515f08123b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:35.166496 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.166462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprqm\" (UniqueName: \"kubernetes.io/projected/fd566afa-c0e6-48c3-894e-a515f08123b9-kube-api-access-qprqm\") pod \"cert-manager-cainjector-559cdd8c87-7x24b\" (UID: \"fd566afa-c0e6-48c3-894e-a515f08123b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:35.166679 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.166527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd566afa-c0e6-48c3-894e-a515f08123b9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7x24b\" (UID: \"fd566afa-c0e6-48c3-894e-a515f08123b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:35.176146 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.176055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd566afa-c0e6-48c3-894e-a515f08123b9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7x24b\" (UID: \"fd566afa-c0e6-48c3-894e-a515f08123b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:35.176260 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.176181 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprqm\" (UniqueName: \"kubernetes.io/projected/fd566afa-c0e6-48c3-894e-a515f08123b9-kube-api-access-qprqm\") pod \"cert-manager-cainjector-559cdd8c87-7x24b\" (UID: \"fd566afa-c0e6-48c3-894e-a515f08123b9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:35.241259 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.241223 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" Aug 04 03:17:35.492730 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:35.492697 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7x24b"] Aug 04 03:17:35.495223 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:17:35.495190 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd566afa_c0e6_48c3_894e_a515f08123b9.slice/crio-85514f7965610aff1ed29c8242b843a42e5ebd2325fdcaded13882ae218a21b3 WatchSource:0}: Error finding container 85514f7965610aff1ed29c8242b843a42e5ebd2325fdcaded13882ae218a21b3: Status 404 returned error can't find the container with id 85514f7965610aff1ed29c8242b843a42e5ebd2325fdcaded13882ae218a21b3 Aug 04 03:17:36.159621 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:36.159593 2568 generic.go:356] "Generic (PLEG): container finished" podID="27bfab85-ad29-4456-93f2-4d5058046514" containerID="a0df1ee536ccd224964de6fae42f0aa94900f27ddd57bd4b93ac707ddb909c1d" exitCode=0 Aug 04 03:17:36.160092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:36.159666 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" event={"ID":"27bfab85-ad29-4456-93f2-4d5058046514","Type":"ContainerDied","Data":"a0df1ee536ccd224964de6fae42f0aa94900f27ddd57bd4b93ac707ddb909c1d"} Aug 04 03:17:36.160862 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:36.160824 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" event={"ID":"fd566afa-c0e6-48c3-894e-a515f08123b9","Type":"ContainerStarted","Data":"85514f7965610aff1ed29c8242b843a42e5ebd2325fdcaded13882ae218a21b3"} Aug 04 03:17:37.169475 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:37.169439 2568 generic.go:356] "Generic (PLEG): container finished" podID="27bfab85-ad29-4456-93f2-4d5058046514" containerID="302f88babee217164549fb4d6e3e9d2515d99effe8dc02657b18be6b90d5f1df" exitCode=0 Aug 04 03:17:37.169946 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:37.169495 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" event={"ID":"27bfab85-ad29-4456-93f2-4d5058046514","Type":"ContainerDied","Data":"302f88babee217164549fb4d6e3e9d2515d99effe8dc02657b18be6b90d5f1df"} Aug 04 03:17:38.402097 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.402074 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:38.495074 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.495046 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdqpb\" (UniqueName: \"kubernetes.io/projected/27bfab85-ad29-4456-93f2-4d5058046514-kube-api-access-rdqpb\") pod \"27bfab85-ad29-4456-93f2-4d5058046514\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " Aug 04 03:17:38.495229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.495145 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-util\") pod \"27bfab85-ad29-4456-93f2-4d5058046514\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " Aug 04 03:17:38.495229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.495173 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-bundle\") pod \"27bfab85-ad29-4456-93f2-4d5058046514\" (UID: \"27bfab85-ad29-4456-93f2-4d5058046514\") " Aug 04 03:17:38.495591 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.495560 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-bundle" (OuterVolumeSpecName: "bundle") pod "27bfab85-ad29-4456-93f2-4d5058046514" (UID: "27bfab85-ad29-4456-93f2-4d5058046514"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:17:38.497179 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.497113 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bfab85-ad29-4456-93f2-4d5058046514-kube-api-access-rdqpb" (OuterVolumeSpecName: "kube-api-access-rdqpb") pod "27bfab85-ad29-4456-93f2-4d5058046514" (UID: "27bfab85-ad29-4456-93f2-4d5058046514"). InnerVolumeSpecName "kube-api-access-rdqpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:17:38.499991 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.499965 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-util" (OuterVolumeSpecName: "util") pod "27bfab85-ad29-4456-93f2-4d5058046514" (UID: "27bfab85-ad29-4456-93f2-4d5058046514"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:17:38.596313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.596276 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-util\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:38.596313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.596303 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27bfab85-ad29-4456-93f2-4d5058046514-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:38.596313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.596313 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdqpb\" (UniqueName: \"kubernetes.io/projected/27bfab85-ad29-4456-93f2-4d5058046514-kube-api-access-rdqpb\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:38.725702 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.725614 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pgnpl"] Aug 04 03:17:38.725925 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.725912 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bfab85-ad29-4456-93f2-4d5058046514" containerName="extract" Aug 04 03:17:38.725925 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.725925 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bfab85-ad29-4456-93f2-4d5058046514" containerName="extract" Aug 04 03:17:38.726034 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.725937 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bfab85-ad29-4456-93f2-4d5058046514" containerName="util" Aug 04 03:17:38.726034 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.725943 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bfab85-ad29-4456-93f2-4d5058046514" containerName="util" Aug 04 03:17:38.726034 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.725951 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27bfab85-ad29-4456-93f2-4d5058046514" containerName="pull" Aug 04 03:17:38.726034 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.725957 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27bfab85-ad29-4456-93f2-4d5058046514" containerName="pull" Aug 04 03:17:38.726034 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.726010 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="27bfab85-ad29-4456-93f2-4d5058046514" containerName="extract" Aug 04 03:17:38.728070 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.728051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:38.730391 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.730368 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sgqxr\"" Aug 04 03:17:38.739859 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.739835 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pgnpl"] Aug 04 03:17:38.899421 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.899390 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4059b662-3bca-4ec3-a0fc-49940f9e8131-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pgnpl\" (UID: \"4059b662-3bca-4ec3-a0fc-49940f9e8131\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:38.899602 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:38.899435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvvm\" (UniqueName: \"kubernetes.io/projected/4059b662-3bca-4ec3-a0fc-49940f9e8131-kube-api-access-smvvm\") pod \"cert-manager-webhook-6486ffd997-pgnpl\" (UID: \"4059b662-3bca-4ec3-a0fc-49940f9e8131\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:39.000951 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.000873 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4059b662-3bca-4ec3-a0fc-49940f9e8131-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pgnpl\" (UID: \"4059b662-3bca-4ec3-a0fc-49940f9e8131\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:39.000951 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.000943 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvvm\" (UniqueName: \"kubernetes.io/projected/4059b662-3bca-4ec3-a0fc-49940f9e8131-kube-api-access-smvvm\") pod \"cert-manager-webhook-6486ffd997-pgnpl\" (UID: \"4059b662-3bca-4ec3-a0fc-49940f9e8131\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:39.009218 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.009195 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4059b662-3bca-4ec3-a0fc-49940f9e8131-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pgnpl\" (UID: \"4059b662-3bca-4ec3-a0fc-49940f9e8131\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:39.009394 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.009374 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvvm\" (UniqueName: \"kubernetes.io/projected/4059b662-3bca-4ec3-a0fc-49940f9e8131-kube-api-access-smvvm\") pod \"cert-manager-webhook-6486ffd997-pgnpl\" (UID: \"4059b662-3bca-4ec3-a0fc-49940f9e8131\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:39.037153 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.037130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:39.160931 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.160885 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pgnpl"] Aug 04 03:17:39.163280 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:17:39.163251 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4059b662_3bca_4ec3_a0fc_49940f9e8131.slice/crio-ab69988abeb29f48616d35fc6e6bb9eae31951d898bd036bbe7e15b4c9aec8b4 WatchSource:0}: Error finding container ab69988abeb29f48616d35fc6e6bb9eae31951d898bd036bbe7e15b4c9aec8b4: Status 404 returned error can't find the container with id ab69988abeb29f48616d35fc6e6bb9eae31951d898bd036bbe7e15b4c9aec8b4 Aug 04 03:17:39.180342 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.180319 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" event={"ID":"27bfab85-ad29-4456-93f2-4d5058046514","Type":"ContainerDied","Data":"77248791ae7714a523cb812cbaef1ace6c9e39348f913a5131ddfa22f436fef0"} Aug 04 03:17:39.180675 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.180339 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2klcr" Aug 04 03:17:39.180675 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.180365 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77248791ae7714a523cb812cbaef1ace6c9e39348f913a5131ddfa22f436fef0" Aug 04 03:17:39.182053 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.182030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" event={"ID":"fd566afa-c0e6-48c3-894e-a515f08123b9","Type":"ContainerStarted","Data":"57f4e3e80b6a1a5a25b753f8a214925dec6d9e6f2333dd5e214f56aa2e1a2921"} Aug 04 03:17:39.183369 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.183348 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" event={"ID":"4059b662-3bca-4ec3-a0fc-49940f9e8131","Type":"ContainerStarted","Data":"ab69988abeb29f48616d35fc6e6bb9eae31951d898bd036bbe7e15b4c9aec8b4"} Aug 04 03:17:39.197840 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:39.197796 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7x24b" podStartSLOduration=2.27545484 podStartE2EDuration="5.197783958s" totalImagesPullingTime="2.922329118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:17:39.196181995 +0000 UTC m=+482.317251338" watchObservedRunningTime="2026-08-04 03:17:39.197783958 +0000 UTC m=+482.318853300" Aug 04 03:17:40.194057 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:40.194019 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" event={"ID":"4059b662-3bca-4ec3-a0fc-49940f9e8131","Type":"ContainerStarted","Data":"8d31d0d5bdd02e077ff36e9e96b0d93d11f891a990f84e9c96621b82eb221349"} Aug 04 03:17:40.194567 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:40.194152 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:40.211652 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:40.211610 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" podStartSLOduration=2.211596853 podStartE2EDuration="2.211596853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:17:40.210123875 +0000 UTC m=+483.331193217" watchObservedRunningTime="2026-08-04 03:17:40.211596853 +0000 UTC m=+483.332666195" Aug 04 03:17:46.201058 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:46.201029 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pgnpl" Aug 04 03:17:53.224053 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.224015 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq"] Aug 04 03:17:53.226446 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.226430 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.229238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.229217 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 03:17:53.229347 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.229218 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-zztjk\"" Aug 04 03:17:53.229347 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.229218 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 03:17:53.243428 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.243399 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq"] Aug 04 03:17:53.314855 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.314812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtzxh\" (UniqueName: \"kubernetes.io/projected/b00496c4-5c80-4e97-981e-0225e8366da1-kube-api-access-wtzxh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.315055 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.314867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.315055 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.314920 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.415918 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.415862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtzxh\" (UniqueName: \"kubernetes.io/projected/b00496c4-5c80-4e97-981e-0225e8366da1-kube-api-access-wtzxh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.416078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.415929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.416078 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.415947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.416324 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.416304 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.416365 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.416310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.423889 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.423859 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtzxh\" (UniqueName: \"kubernetes.io/projected/b00496c4-5c80-4e97-981e-0225e8366da1-kube-api-access-wtzxh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.536826 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.536738 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:53.664002 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:53.663975 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq"] Aug 04 03:17:53.666264 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:17:53.666239 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00496c4_5c80_4e97_981e_0225e8366da1.slice/crio-cfeb2e3c29c49e12bf01cdb3247ad80445ee4a5b97741c0d716642ab47a5d298 WatchSource:0}: Error finding container cfeb2e3c29c49e12bf01cdb3247ad80445ee4a5b97741c0d716642ab47a5d298: Status 404 returned error can't find the container with id cfeb2e3c29c49e12bf01cdb3247ad80445ee4a5b97741c0d716642ab47a5d298 Aug 04 03:17:54.257322 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:54.257287 2568 generic.go:356] "Generic (PLEG): container finished" podID="b00496c4-5c80-4e97-981e-0225e8366da1" containerID="3433b4164ac1d8f9977aee964d7f89d65df03aa0f4e5143b656e71fec1117fe8" exitCode=0 Aug 04 03:17:54.257691 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:54.257380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" event={"ID":"b00496c4-5c80-4e97-981e-0225e8366da1","Type":"ContainerDied","Data":"3433b4164ac1d8f9977aee964d7f89d65df03aa0f4e5143b656e71fec1117fe8"} Aug 04 03:17:54.257691 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:54.257432 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" event={"ID":"b00496c4-5c80-4e97-981e-0225e8366da1","Type":"ContainerStarted","Data":"cfeb2e3c29c49e12bf01cdb3247ad80445ee4a5b97741c0d716642ab47a5d298"} Aug 04 03:17:55.263289 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:55.263251 2568 generic.go:356] "Generic (PLEG): container finished" podID="b00496c4-5c80-4e97-981e-0225e8366da1" containerID="ca9d5e82410f0c13f7f82749cf2dea65134485539f3a78725cab76b0f40dbd96" exitCode=0 Aug 04 03:17:55.263755 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:55.263334 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" event={"ID":"b00496c4-5c80-4e97-981e-0225e8366da1","Type":"ContainerDied","Data":"ca9d5e82410f0c13f7f82749cf2dea65134485539f3a78725cab76b0f40dbd96"} Aug 04 03:17:56.269250 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:56.269215 2568 generic.go:356] "Generic (PLEG): container finished" podID="b00496c4-5c80-4e97-981e-0225e8366da1" containerID="ea0475f4aa33bc1a0e30cbfd0badbd67f811ae61207336022fb06912023e0393" exitCode=0 Aug 04 03:17:56.269631 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:56.269282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" event={"ID":"b00496c4-5c80-4e97-981e-0225e8366da1","Type":"ContainerDied","Data":"ea0475f4aa33bc1a0e30cbfd0badbd67f811ae61207336022fb06912023e0393"} Aug 04 03:17:57.405285 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.405258 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:57.542160 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.542077 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-util\") pod \"b00496c4-5c80-4e97-981e-0225e8366da1\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " Aug 04 03:17:57.542160 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.542131 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-bundle\") pod \"b00496c4-5c80-4e97-981e-0225e8366da1\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " Aug 04 03:17:57.542339 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.542172 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtzxh\" (UniqueName: \"kubernetes.io/projected/b00496c4-5c80-4e97-981e-0225e8366da1-kube-api-access-wtzxh\") pod \"b00496c4-5c80-4e97-981e-0225e8366da1\" (UID: \"b00496c4-5c80-4e97-981e-0225e8366da1\") " Aug 04 03:17:57.542933 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.542878 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-bundle" (OuterVolumeSpecName: "bundle") pod "b00496c4-5c80-4e97-981e-0225e8366da1" (UID: "b00496c4-5c80-4e97-981e-0225e8366da1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:17:57.544256 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.544227 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00496c4-5c80-4e97-981e-0225e8366da1-kube-api-access-wtzxh" (OuterVolumeSpecName: "kube-api-access-wtzxh") pod "b00496c4-5c80-4e97-981e-0225e8366da1" (UID: "b00496c4-5c80-4e97-981e-0225e8366da1"). InnerVolumeSpecName "kube-api-access-wtzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:17:57.547772 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.547748 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-util" (OuterVolumeSpecName: "util") pod "b00496c4-5c80-4e97-981e-0225e8366da1" (UID: "b00496c4-5c80-4e97-981e-0225e8366da1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:17:57.642960 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.642926 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-util\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:57.642960 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.642954 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b00496c4-5c80-4e97-981e-0225e8366da1-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:57.642960 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:57.642964 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtzxh\" (UniqueName: \"kubernetes.io/projected/b00496c4-5c80-4e97-981e-0225e8366da1-kube-api-access-wtzxh\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:17:58.280089 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:58.280053 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" event={"ID":"b00496c4-5c80-4e97-981e-0225e8366da1","Type":"ContainerDied","Data":"cfeb2e3c29c49e12bf01cdb3247ad80445ee4a5b97741c0d716642ab47a5d298"} Aug 04 03:17:58.280089 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:58.280084 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmmnwq" Aug 04 03:17:58.280288 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:17:58.280105 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfeb2e3c29c49e12bf01cdb3247ad80445ee4a5b97741c0d716642ab47a5d298" Aug 04 03:18:04.107803 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.107768 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x"] Aug 04 03:18:04.108244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.108063 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b00496c4-5c80-4e97-981e-0225e8366da1" containerName="pull" Aug 04 03:18:04.108244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.108074 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b00496c4-5c80-4e97-981e-0225e8366da1" containerName="pull" Aug 04 03:18:04.108244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.108085 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b00496c4-5c80-4e97-981e-0225e8366da1" containerName="util" Aug 04 03:18:04.108244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.108091 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b00496c4-5c80-4e97-981e-0225e8366da1" containerName="util" Aug 04 03:18:04.108244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.108102 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b00496c4-5c80-4e97-981e-0225e8366da1" containerName="extract" Aug 04 03:18:04.108244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.108108 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b00496c4-5c80-4e97-981e-0225e8366da1" containerName="extract" Aug 04 03:18:04.108244 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.108164 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b00496c4-5c80-4e97-981e-0225e8366da1" containerName="extract" Aug 04 03:18:04.110920 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.110886 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.114080 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.114050 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 03:18:04.114208 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.114080 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 03:18:04.114208 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.114154 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 03:18:04.114362 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.114347 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cbjbp\"" Aug 04 03:18:04.114415 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.114380 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 03:18:04.126878 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.126856 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x"] Aug 04 03:18:04.186400 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.186370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37f7b958-9493-41bd-b931-9451f73b676f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.186532 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.186410 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37f7b958-9493-41bd-b931-9451f73b676f-manifests\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.186532 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.186433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37f7b958-9493-41bd-b931-9451f73b676f-tmp\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.186532 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.186459 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvdm\" (UniqueName: \"kubernetes.io/projected/37f7b958-9493-41bd-b931-9451f73b676f-kube-api-access-hpvdm\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.186532 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.186511 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37f7b958-9493-41bd-b931-9451f73b676f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.287098 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.287069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37f7b958-9493-41bd-b931-9451f73b676f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.287229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.287108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37f7b958-9493-41bd-b931-9451f73b676f-manifests\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.287229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.287134 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37f7b958-9493-41bd-b931-9451f73b676f-tmp\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.287229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.287154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvdm\" (UniqueName: \"kubernetes.io/projected/37f7b958-9493-41bd-b931-9451f73b676f-kube-api-access-hpvdm\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.287229 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.287208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37f7b958-9493-41bd-b931-9451f73b676f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.287517 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.287494 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/37f7b958-9493-41bd-b931-9451f73b676f-manifests\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.289394 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.289369 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37f7b958-9493-41bd-b931-9451f73b676f-tmp\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.289613 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.289597 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/37f7b958-9493-41bd-b931-9451f73b676f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.289652 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.289636 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/37f7b958-9493-41bd-b931-9451f73b676f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.299243 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.299219 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvdm\" (UniqueName: \"kubernetes.io/projected/37f7b958-9493-41bd-b931-9451f73b676f-kube-api-access-hpvdm\") pod \"opendatahub-operator-controller-manager-6fb59c9fb6-sng5x\" (UID: \"37f7b958-9493-41bd-b931-9451f73b676f\") " pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.421178 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.421146 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:04.555499 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:04.555469 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x"] Aug 04 03:18:04.558834 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:18:04.558809 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f7b958_9493_41bd_b931_9451f73b676f.slice/crio-7e1cef2b53f4a97f2a9bd047d80fe7c6d2a92a73bc6c5f72cd3447181955bc3b WatchSource:0}: Error finding container 7e1cef2b53f4a97f2a9bd047d80fe7c6d2a92a73bc6c5f72cd3447181955bc3b: Status 404 returned error can't find the container with id 7e1cef2b53f4a97f2a9bd047d80fe7c6d2a92a73bc6c5f72cd3447181955bc3b Aug 04 03:18:05.318762 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:05.318719 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" event={"ID":"37f7b958-9493-41bd-b931-9451f73b676f","Type":"ContainerStarted","Data":"7e1cef2b53f4a97f2a9bd047d80fe7c6d2a92a73bc6c5f72cd3447181955bc3b"} Aug 04 03:18:08.337245 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:08.337208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" event={"ID":"37f7b958-9493-41bd-b931-9451f73b676f","Type":"ContainerStarted","Data":"cfdfccd714ea7ef13bcb60901607b308d26717069b1b089900fc84897c5d9d1f"} Aug 04 03:18:09.343177 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:09.343139 2568 generic.go:356] "Generic (PLEG): container finished" podID="37f7b958-9493-41bd-b931-9451f73b676f" containerID="cfdfccd714ea7ef13bcb60901607b308d26717069b1b089900fc84897c5d9d1f" exitCode=0 Aug 04 03:18:09.343545 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:09.343233 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" event={"ID":"37f7b958-9493-41bd-b931-9451f73b676f","Type":"ContainerDied","Data":"cfdfccd714ea7ef13bcb60901607b308d26717069b1b089900fc84897c5d9d1f"} Aug 04 03:18:10.349848 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:10.349811 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" event={"ID":"37f7b958-9493-41bd-b931-9451f73b676f","Type":"ContainerStarted","Data":"b46f7156ae298749557fe2d1e009bad8ee2bd932499019f16c3663c8ecd828e0"} Aug 04 03:18:10.350272 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:10.350082 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:10.373858 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:10.373811 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" podStartSLOduration=2.084446735 podStartE2EDuration="6.373797195s" totalImagesPullingTime="3.241056126s" totalInitContainerRuntime="1.048294334s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:18:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:18:10.371153774 +0000 UTC m=+513.492223133" watchObservedRunningTime="2026-08-04 03:18:10.373797195 +0000 UTC m=+513.494866536" Aug 04 03:18:21.356466 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:21.356436 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6fb59c9fb6-sng5x" Aug 04 03:18:23.888612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.888578 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2rtv6"] Aug 04 03:18:23.897447 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.897426 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:23.900323 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.900303 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hwj7d\"" Aug 04 03:18:23.900493 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.900477 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 03:18:23.906071 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.906048 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2rtv6"] Aug 04 03:18:23.938438 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.938412 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be52a1d3-c577-422e-bb79-7852f540c3f0-cert\") pod \"odh-model-controller-7998c67dc4-2rtv6\" (UID: \"be52a1d3-c577-422e-bb79-7852f540c3f0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:23.938577 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.938452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vrz\" (UniqueName: \"kubernetes.io/projected/be52a1d3-c577-422e-bb79-7852f540c3f0-kube-api-access-l9vrz\") pod \"odh-model-controller-7998c67dc4-2rtv6\" (UID: \"be52a1d3-c577-422e-bb79-7852f540c3f0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:23.985301 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.985273 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl"] Aug 04 03:18:23.988431 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.988416 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:23.991284 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.991210 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 03:18:23.991284 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.991228 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 03:18:23.991284 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.991228 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-622s4\"" Aug 04 03:18:23.991533 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.991276 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 03:18:23.991533 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.991276 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 03:18:23.991533 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.991345 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 03:18:23.998764 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:23.998742 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl"] Aug 04 03:18:24.039612 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.039585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d192bd50-e074-452e-ad02-c67392401775-manager-config\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.039739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.039618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d192bd50-e074-452e-ad02-c67392401775-cert\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.039739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.039649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be52a1d3-c577-422e-bb79-7852f540c3f0-cert\") pod \"odh-model-controller-7998c67dc4-2rtv6\" (UID: \"be52a1d3-c577-422e-bb79-7852f540c3f0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:24.039739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.039709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vrz\" (UniqueName: \"kubernetes.io/projected/be52a1d3-c577-422e-bb79-7852f540c3f0-kube-api-access-l9vrz\") pod \"odh-model-controller-7998c67dc4-2rtv6\" (UID: \"be52a1d3-c577-422e-bb79-7852f540c3f0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:24.039739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.039737 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwnk\" (UniqueName: \"kubernetes.io/projected/d192bd50-e074-452e-ad02-c67392401775-kube-api-access-ttwnk\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.039967 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:18:24.039749 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 04 03:18:24.039967 ip-10-0-130-28 kubenswrapper[2568]: E0804 03:18:24.039804 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be52a1d3-c577-422e-bb79-7852f540c3f0-cert podName:be52a1d3-c577-422e-bb79-7852f540c3f0 nodeName:}" failed. No retries permitted until 2026-08-04 03:18:24.539785687 +0000 UTC m=+527.660855021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/be52a1d3-c577-422e-bb79-7852f540c3f0-cert") pod "odh-model-controller-7998c67dc4-2rtv6" (UID: "be52a1d3-c577-422e-bb79-7852f540c3f0") : secret "odh-model-controller-webhook-cert" not found Aug 04 03:18:24.039967 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.039824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d192bd50-e074-452e-ad02-c67392401775-metrics-cert\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.058153 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.058130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vrz\" (UniqueName: \"kubernetes.io/projected/be52a1d3-c577-422e-bb79-7852f540c3f0-kube-api-access-l9vrz\") pod \"odh-model-controller-7998c67dc4-2rtv6\" (UID: \"be52a1d3-c577-422e-bb79-7852f540c3f0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:24.140805 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.140726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d192bd50-e074-452e-ad02-c67392401775-metrics-cert\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.140805 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.140795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d192bd50-e074-452e-ad02-c67392401775-manager-config\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.140992 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.140823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d192bd50-e074-452e-ad02-c67392401775-cert\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.140992 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.140877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwnk\" (UniqueName: \"kubernetes.io/projected/d192bd50-e074-452e-ad02-c67392401775-kube-api-access-ttwnk\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.141414 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.141392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d192bd50-e074-452e-ad02-c67392401775-manager-config\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.143288 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.143267 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d192bd50-e074-452e-ad02-c67392401775-cert\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.143429 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.143410 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d192bd50-e074-452e-ad02-c67392401775-metrics-cert\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.148501 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.148481 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwnk\" (UniqueName: \"kubernetes.io/projected/d192bd50-e074-452e-ad02-c67392401775-kube-api-access-ttwnk\") pod \"lws-controller-manager-6488bdc59d-894hl\" (UID: \"d192bd50-e074-452e-ad02-c67392401775\") " pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.297392 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.297362 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:24.428076 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.428010 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl"] Aug 04 03:18:24.430244 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:18:24.430220 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd192bd50_e074_452e_ad02_c67392401775.slice/crio-7fdc6d392976a853da5449a2e6d75a67fdb1f442469c89fd2e455a4a18683767 WatchSource:0}: Error finding container 7fdc6d392976a853da5449a2e6d75a67fdb1f442469c89fd2e455a4a18683767: Status 404 returned error can't find the container with id 7fdc6d392976a853da5449a2e6d75a67fdb1f442469c89fd2e455a4a18683767 Aug 04 03:18:24.544336 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.544305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be52a1d3-c577-422e-bb79-7852f540c3f0-cert\") pod \"odh-model-controller-7998c67dc4-2rtv6\" (UID: \"be52a1d3-c577-422e-bb79-7852f540c3f0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:24.546632 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.546613 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be52a1d3-c577-422e-bb79-7852f540c3f0-cert\") pod \"odh-model-controller-7998c67dc4-2rtv6\" (UID: \"be52a1d3-c577-422e-bb79-7852f540c3f0\") " pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:24.808936 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.808814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:24.936738 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:24.936714 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2rtv6"] Aug 04 03:18:24.938638 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:18:24.938610 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe52a1d3_c577_422e_bb79_7852f540c3f0.slice/crio-208a083dd8f4c20e119c9d4fd3210be8113bee62260e13aed0a25c713c692963 WatchSource:0}: Error finding container 208a083dd8f4c20e119c9d4fd3210be8113bee62260e13aed0a25c713c692963: Status 404 returned error can't find the container with id 208a083dd8f4c20e119c9d4fd3210be8113bee62260e13aed0a25c713c692963 Aug 04 03:18:25.429829 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:25.429788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" event={"ID":"be52a1d3-c577-422e-bb79-7852f540c3f0","Type":"ContainerStarted","Data":"208a083dd8f4c20e119c9d4fd3210be8113bee62260e13aed0a25c713c692963"} Aug 04 03:18:25.431090 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:25.431065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" event={"ID":"d192bd50-e074-452e-ad02-c67392401775","Type":"ContainerStarted","Data":"7fdc6d392976a853da5449a2e6d75a67fdb1f442469c89fd2e455a4a18683767"} Aug 04 03:18:26.439981 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:26.439941 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" event={"ID":"d192bd50-e074-452e-ad02-c67392401775","Type":"ContainerStarted","Data":"f31a3f0f3959b0282e56472cee29d8552f032033952bd60172a8a74b482339a6"} Aug 04 03:18:26.440410 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:26.440086 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:26.457475 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:26.457415 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" podStartSLOduration=1.642469217 podStartE2EDuration="3.457395783s" totalImagesPullingTime="1.814926566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:18:26.454946738 +0000 UTC m=+529.576016081" watchObservedRunningTime="2026-08-04 03:18:26.457395783 +0000 UTC m=+529.578465126" Aug 04 03:18:28.449882 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:28.449850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" event={"ID":"be52a1d3-c577-422e-bb79-7852f540c3f0","Type":"ContainerStarted","Data":"b13cf3148e8c99a7fd626cc3aa7164bef7f210b2adaf9b2b584c06e6320980d0"} Aug 04 03:18:28.450242 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:28.449927 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:18:28.466527 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:28.466471 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" podStartSLOduration=2.3793832569999998 podStartE2EDuration="5.466452471s" totalImagesPullingTime="3.087069214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:18:28.465054855 +0000 UTC m=+531.586124199" watchObservedRunningTime="2026-08-04 03:18:28.466452471 +0000 UTC m=+531.587521814" Aug 04 03:18:32.300273 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.300239 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp"] Aug 04 03:18:32.303494 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.303478 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.306377 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.306354 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 03:18:32.306489 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.306384 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 03:18:32.306489 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.306403 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 03:18:32.306489 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.306354 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 03:18:32.306667 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.306640 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9prf4\"" Aug 04 03:18:32.315199 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.315181 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp"] Aug 04 03:18:32.413049 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.413016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99443321-7b05-4242-a9d5-1773be5407f3-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.413238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.413059 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.413238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.413089 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99443321-7b05-4242-a9d5-1773be5407f3-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.413238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.413156 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.413238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.413186 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6p6d\" (UniqueName: \"kubernetes.io/projected/99443321-7b05-4242-a9d5-1773be5407f3-kube-api-access-r6p6d\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.413238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.413218 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.413238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.413235 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99443321-7b05-4242-a9d5-1773be5407f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.513905 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.513870 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.514077 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.513926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99443321-7b05-4242-a9d5-1773be5407f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.514077 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.513954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99443321-7b05-4242-a9d5-1773be5407f3-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.514077 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.513978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.514077 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.514004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99443321-7b05-4242-a9d5-1773be5407f3-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.514238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.514204 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.514290 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.514248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6p6d\" (UniqueName: \"kubernetes.io/projected/99443321-7b05-4242-a9d5-1773be5407f3-kube-api-access-r6p6d\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.514744 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.514716 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/99443321-7b05-4242-a9d5-1773be5407f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.516495 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.516450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.516610 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.516578 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.516677 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.516623 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/99443321-7b05-4242-a9d5-1773be5407f3-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.516827 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.516807 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/99443321-7b05-4242-a9d5-1773be5407f3-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.522286 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.522263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99443321-7b05-4242-a9d5-1773be5407f3-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.523406 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.523384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6p6d\" (UniqueName: \"kubernetes.io/projected/99443321-7b05-4242-a9d5-1773be5407f3-kube-api-access-r6p6d\") pod \"istiod-openshift-gateway-5677ccb49b-6p4fp\" (UID: \"99443321-7b05-4242-a9d5-1773be5407f3\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.612676 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.612601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:32.773625 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:32.773597 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp"] Aug 04 03:18:32.775280 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:18:32.775255 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99443321_7b05_4242_a9d5_1773be5407f3.slice/crio-03e6e08f06d3a04515fb6e257bd30c6ab29da1dfc2845699b699812e98e3046d WatchSource:0}: Error finding container 03e6e08f06d3a04515fb6e257bd30c6ab29da1dfc2845699b699812e98e3046d: Status 404 returned error can't find the container with id 03e6e08f06d3a04515fb6e257bd30c6ab29da1dfc2845699b699812e98e3046d Aug 04 03:18:33.474603 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:33.474570 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" event={"ID":"99443321-7b05-4242-a9d5-1773be5407f3","Type":"ContainerStarted","Data":"03e6e08f06d3a04515fb6e257bd30c6ab29da1dfc2845699b699812e98e3046d"} Aug 04 03:18:35.365161 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:35.365125 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 03:18:35.365433 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:35.365191 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 03:18:35.487241 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:35.487208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" event={"ID":"99443321-7b05-4242-a9d5-1773be5407f3","Type":"ContainerStarted","Data":"42e4065545cacfa089f6df5d0bbf14bcb598f32c20f9d1b0f3cd19d595d9a077"} Aug 04 03:18:35.487397 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:35.487333 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:35.523808 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:35.523760 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" podStartSLOduration=0.936144819 podStartE2EDuration="3.523745722s" totalImagesPullingTime="2.587600903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:18:35.522349721 +0000 UTC m=+538.643419063" watchObservedRunningTime="2026-08-04 03:18:35.523745722 +0000 UTC m=+538.644815063" Aug 04 03:18:36.493601 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:36.493571 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-6p4fp" Aug 04 03:18:37.445806 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:37.445777 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6488bdc59d-894hl" Aug 04 03:18:39.458506 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:18:39.458478 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2rtv6" Aug 04 03:19:39.644249 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.644142 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:19:39.648446 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.646772 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:19:39.915211 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.915184 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n"] Aug 04 03:19:39.918741 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.918724 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:39.921695 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.921677 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 03:19:39.921778 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.921677 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vjhhf\"" Aug 04 03:19:39.922551 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.922519 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 03:19:39.926422 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:39.926402 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n"] Aug 04 03:19:40.005603 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.005571 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.005603 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.005602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.005822 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.005636 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtpmm\" (UniqueName: \"kubernetes.io/projected/36ad4170-0edc-490f-beac-3c807ea2da07-kube-api-access-xtpmm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.106403 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.106367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.106403 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.106403 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.106641 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.106451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtpmm\" (UniqueName: \"kubernetes.io/projected/36ad4170-0edc-490f-beac-3c807ea2da07-kube-api-access-xtpmm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.106828 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.106803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.106923 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.106816 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.115391 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.115368 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtpmm\" (UniqueName: \"kubernetes.io/projected/36ad4170-0edc-490f-beac-3c807ea2da07-kube-api-access-xtpmm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.229107 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.229020 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:40.318594 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.318561 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv"] Aug 04 03:19:40.322251 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.322233 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.329910 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.329866 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv"] Aug 04 03:19:40.366386 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.366363 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n"] Aug 04 03:19:40.368311 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:19:40.368284 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ad4170_0edc_490f_beac_3c807ea2da07.slice/crio-a4002730d3b0807fac8372232f374bd5b4511558d58b5660873cc16338908d3d WatchSource:0}: Error finding container a4002730d3b0807fac8372232f374bd5b4511558d58b5660873cc16338908d3d: Status 404 returned error can't find the container with id a4002730d3b0807fac8372232f374bd5b4511558d58b5660873cc16338908d3d Aug 04 03:19:40.409196 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.409172 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.409296 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.409232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.409296 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.409255 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hk4\" (UniqueName: \"kubernetes.io/projected/e724473a-29da-4b06-aeb5-a8929046f137-kube-api-access-42hk4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.510232 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.510158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.510232 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.510203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hk4\" (UniqueName: \"kubernetes.io/projected/e724473a-29da-4b06-aeb5-a8929046f137-kube-api-access-42hk4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.510432 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.510307 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.510488 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.510474 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.510621 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.510602 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.518413 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.518392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hk4\" (UniqueName: \"kubernetes.io/projected/e724473a-29da-4b06-aeb5-a8929046f137-kube-api-access-42hk4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.635313 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.635282 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:40.762587 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.762567 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv"] Aug 04 03:19:40.764684 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:19:40.764654 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode724473a_29da_4b06_aeb5_a8929046f137.slice/crio-e860cb9fb19bf0a65d0ee0054162c1f93f2607f47ee6e63681a8dca93acd8638 WatchSource:0}: Error finding container e860cb9fb19bf0a65d0ee0054162c1f93f2607f47ee6e63681a8dca93acd8638: Status 404 returned error can't find the container with id e860cb9fb19bf0a65d0ee0054162c1f93f2607f47ee6e63681a8dca93acd8638 Aug 04 03:19:40.791159 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.791137 2568 generic.go:356] "Generic (PLEG): container finished" podID="36ad4170-0edc-490f-beac-3c807ea2da07" containerID="7faa9cc17c96fa765133a2cfbea8cdf3fac1f44fa8ba6020cff23b3ff5e8debe" exitCode=0 Aug 04 03:19:40.791248 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.791223 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" event={"ID":"36ad4170-0edc-490f-beac-3c807ea2da07","Type":"ContainerDied","Data":"7faa9cc17c96fa765133a2cfbea8cdf3fac1f44fa8ba6020cff23b3ff5e8debe"} Aug 04 03:19:40.791297 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.791274 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" event={"ID":"36ad4170-0edc-490f-beac-3c807ea2da07","Type":"ContainerStarted","Data":"a4002730d3b0807fac8372232f374bd5b4511558d58b5660873cc16338908d3d"} Aug 04 03:19:40.791700 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.791683 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 03:19:40.792316 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.792298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" event={"ID":"e724473a-29da-4b06-aeb5-a8929046f137","Type":"ContainerStarted","Data":"e860cb9fb19bf0a65d0ee0054162c1f93f2607f47ee6e63681a8dca93acd8638"} Aug 04 03:19:40.915479 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.915448 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd"] Aug 04 03:19:40.918795 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.918779 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:40.926780 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:40.926757 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd"] Aug 04 03:19:41.013097 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.013028 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdgtc\" (UniqueName: \"kubernetes.io/projected/38e874d0-ae52-409a-9ac3-21243b5aa4d6-kube-api-access-qdgtc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.013097 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.013075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.013268 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.013096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.113945 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.113887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdgtc\" (UniqueName: \"kubernetes.io/projected/38e874d0-ae52-409a-9ac3-21243b5aa4d6-kube-api-access-qdgtc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.114098 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.113984 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.114098 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.114010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.114374 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.114356 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.114414 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.114384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.121935 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.121887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdgtc\" (UniqueName: \"kubernetes.io/projected/38e874d0-ae52-409a-9ac3-21243b5aa4d6-kube-api-access-qdgtc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.228464 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.228429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:41.355471 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.355442 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd"] Aug 04 03:19:41.383943 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:19:41.383911 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e874d0_ae52_409a_9ac3_21243b5aa4d6.slice/crio-f8700858650339228f23625a9d59adba3820b86a59b9a51085c222b383e30d4c WatchSource:0}: Error finding container f8700858650339228f23625a9d59adba3820b86a59b9a51085c222b383e30d4c: Status 404 returned error can't find the container with id f8700858650339228f23625a9d59adba3820b86a59b9a51085c222b383e30d4c Aug 04 03:19:41.515113 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.515086 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b"] Aug 04 03:19:41.518580 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.518563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.525927 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.525905 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b"] Aug 04 03:19:41.617557 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.617526 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.617710 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.617575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.617710 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.617685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7j9n\" (UniqueName: \"kubernetes.io/projected/43031ab5-11a6-4b69-b581-2813b4db045f-kube-api-access-g7j9n\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.718886 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.718848 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.719067 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.718953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.719067 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.719018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7j9n\" (UniqueName: \"kubernetes.io/projected/43031ab5-11a6-4b69-b581-2813b4db045f-kube-api-access-g7j9n\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.719263 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.719242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.719345 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.719324 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.728765 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.728745 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7j9n\" (UniqueName: \"kubernetes.io/projected/43031ab5-11a6-4b69-b581-2813b4db045f-kube-api-access-g7j9n\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.798831 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.798747 2568 generic.go:356] "Generic (PLEG): container finished" podID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerID="6dac9568803abaf6b9c78bf816a10bd8dfdd7b2de5d5acb12cdfb807bcdd1fc4" exitCode=0 Aug 04 03:19:41.799251 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.798832 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" event={"ID":"38e874d0-ae52-409a-9ac3-21243b5aa4d6","Type":"ContainerDied","Data":"6dac9568803abaf6b9c78bf816a10bd8dfdd7b2de5d5acb12cdfb807bcdd1fc4"} Aug 04 03:19:41.799251 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.798928 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" event={"ID":"38e874d0-ae52-409a-9ac3-21243b5aa4d6","Type":"ContainerStarted","Data":"f8700858650339228f23625a9d59adba3820b86a59b9a51085c222b383e30d4c"} Aug 04 03:19:41.800740 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.800721 2568 generic.go:356] "Generic (PLEG): container finished" podID="36ad4170-0edc-490f-beac-3c807ea2da07" containerID="ff3d65a2f3750e39261d486e273f922e5bc53e9d20f3a8591d96e95a2628c692" exitCode=0 Aug 04 03:19:41.800835 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.800789 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" event={"ID":"36ad4170-0edc-490f-beac-3c807ea2da07","Type":"ContainerDied","Data":"ff3d65a2f3750e39261d486e273f922e5bc53e9d20f3a8591d96e95a2628c692"} Aug 04 03:19:41.802414 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.802391 2568 generic.go:356] "Generic (PLEG): container finished" podID="e724473a-29da-4b06-aeb5-a8929046f137" containerID="7839c55e3f7f2b8b0ecc42de590ac05d343f1a732592bdf23170168d9c2aba97" exitCode=0 Aug 04 03:19:41.802492 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.802432 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" event={"ID":"e724473a-29da-4b06-aeb5-a8929046f137","Type":"ContainerDied","Data":"7839c55e3f7f2b8b0ecc42de590ac05d343f1a732592bdf23170168d9c2aba97"} Aug 04 03:19:41.839505 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.839486 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:41.982769 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:41.982741 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b"] Aug 04 03:19:41.984670 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:19:41.984641 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43031ab5_11a6_4b69_b581_2813b4db045f.slice/crio-2c9c6216a58f70867e0c602b4173138e8d468049b06ac23acf53302dabbcaa1d WatchSource:0}: Error finding container 2c9c6216a58f70867e0c602b4173138e8d468049b06ac23acf53302dabbcaa1d: Status 404 returned error can't find the container with id 2c9c6216a58f70867e0c602b4173138e8d468049b06ac23acf53302dabbcaa1d Aug 04 03:19:42.808560 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.808531 2568 generic.go:356] "Generic (PLEG): container finished" podID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerID="56d2421df65e05685a396efa7fe5e5f57e57c7b8f399fe4851f980f3b3e3707c" exitCode=0 Aug 04 03:19:42.808946 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.808617 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" event={"ID":"38e874d0-ae52-409a-9ac3-21243b5aa4d6","Type":"ContainerDied","Data":"56d2421df65e05685a396efa7fe5e5f57e57c7b8f399fe4851f980f3b3e3707c"} Aug 04 03:19:42.810138 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.810115 2568 generic.go:356] "Generic (PLEG): container finished" podID="43031ab5-11a6-4b69-b581-2813b4db045f" containerID="ef3333275c01d5ca3881935b24247441a0d628fa2f6292fb859fab8345ca3bca" exitCode=0 Aug 04 03:19:42.810250 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.810211 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" event={"ID":"43031ab5-11a6-4b69-b581-2813b4db045f","Type":"ContainerDied","Data":"ef3333275c01d5ca3881935b24247441a0d628fa2f6292fb859fab8345ca3bca"} Aug 04 03:19:42.810379 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.810252 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" event={"ID":"43031ab5-11a6-4b69-b581-2813b4db045f","Type":"ContainerStarted","Data":"2c9c6216a58f70867e0c602b4173138e8d468049b06ac23acf53302dabbcaa1d"} Aug 04 03:19:42.812532 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.812514 2568 generic.go:356] "Generic (PLEG): container finished" podID="36ad4170-0edc-490f-beac-3c807ea2da07" containerID="65e56b370ba16d8a91ff1b88caa682e11c3bcc9fdfcd55b95550905cc31f3d00" exitCode=0 Aug 04 03:19:42.812626 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.812583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" event={"ID":"36ad4170-0edc-490f-beac-3c807ea2da07","Type":"ContainerDied","Data":"65e56b370ba16d8a91ff1b88caa682e11c3bcc9fdfcd55b95550905cc31f3d00"} Aug 04 03:19:42.814471 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.814453 2568 generic.go:356] "Generic (PLEG): container finished" podID="e724473a-29da-4b06-aeb5-a8929046f137" containerID="540be622a09cba2a634dafc88bafbdb2c4cc9cc3c66bb02d0b757acdba61a11e" exitCode=0 Aug 04 03:19:42.814560 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:42.814501 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" event={"ID":"e724473a-29da-4b06-aeb5-a8929046f137","Type":"ContainerDied","Data":"540be622a09cba2a634dafc88bafbdb2c4cc9cc3c66bb02d0b757acdba61a11e"} Aug 04 03:19:43.821148 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:43.821115 2568 generic.go:356] "Generic (PLEG): container finished" podID="e724473a-29da-4b06-aeb5-a8929046f137" containerID="80671cc7c07a74a97cdb260472e49b060efe81ffcde323472562df3398a6c0b6" exitCode=0 Aug 04 03:19:43.821515 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:43.821191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" event={"ID":"e724473a-29da-4b06-aeb5-a8929046f137","Type":"ContainerDied","Data":"80671cc7c07a74a97cdb260472e49b060efe81ffcde323472562df3398a6c0b6"} Aug 04 03:19:43.823167 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:43.823144 2568 generic.go:356] "Generic (PLEG): container finished" podID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerID="d7ceb6fe08212846e8349fd6f24c618f5566f7dd43bf9051593555b41082a5bb" exitCode=0 Aug 04 03:19:43.823270 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:43.823213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" event={"ID":"38e874d0-ae52-409a-9ac3-21243b5aa4d6","Type":"ContainerDied","Data":"d7ceb6fe08212846e8349fd6f24c618f5566f7dd43bf9051593555b41082a5bb"} Aug 04 03:19:43.824919 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:43.824873 2568 generic.go:356] "Generic (PLEG): container finished" podID="43031ab5-11a6-4b69-b581-2813b4db045f" containerID="eea6892dcee30cdb531008c2c9ba434961c230d62bf7b32285e27118c160963b" exitCode=0 Aug 04 03:19:43.825017 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:43.824990 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" event={"ID":"43031ab5-11a6-4b69-b581-2813b4db045f","Type":"ContainerDied","Data":"eea6892dcee30cdb531008c2c9ba434961c230d62bf7b32285e27118c160963b"} Aug 04 03:19:43.963242 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:43.963219 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:44.036409 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.036381 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-util\") pod \"36ad4170-0edc-490f-beac-3c807ea2da07\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " Aug 04 03:19:44.036565 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.036427 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-bundle\") pod \"36ad4170-0edc-490f-beac-3c807ea2da07\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " Aug 04 03:19:44.036565 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.036453 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtpmm\" (UniqueName: \"kubernetes.io/projected/36ad4170-0edc-490f-beac-3c807ea2da07-kube-api-access-xtpmm\") pod \"36ad4170-0edc-490f-beac-3c807ea2da07\" (UID: \"36ad4170-0edc-490f-beac-3c807ea2da07\") " Aug 04 03:19:44.036838 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.036815 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-bundle" (OuterVolumeSpecName: "bundle") pod "36ad4170-0edc-490f-beac-3c807ea2da07" (UID: "36ad4170-0edc-490f-beac-3c807ea2da07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:44.038521 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.038501 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ad4170-0edc-490f-beac-3c807ea2da07-kube-api-access-xtpmm" (OuterVolumeSpecName: "kube-api-access-xtpmm") pod "36ad4170-0edc-490f-beac-3c807ea2da07" (UID: "36ad4170-0edc-490f-beac-3c807ea2da07"). InnerVolumeSpecName "kube-api-access-xtpmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:19:44.041823 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.041800 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-util" (OuterVolumeSpecName: "util") pod "36ad4170-0edc-490f-beac-3c807ea2da07" (UID: "36ad4170-0edc-490f-beac-3c807ea2da07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:44.137055 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.137027 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:44.137166 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.137056 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtpmm\" (UniqueName: \"kubernetes.io/projected/36ad4170-0edc-490f-beac-3c807ea2da07-kube-api-access-xtpmm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:44.137166 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.137066 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/36ad4170-0edc-490f-beac-3c807ea2da07-util\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:44.832215 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.832179 2568 generic.go:356] "Generic (PLEG): container finished" podID="43031ab5-11a6-4b69-b581-2813b4db045f" containerID="3dda88e5691cbaadce2f96abe96ddc097d19835cd1c513c906e8fc18ef8f6396" exitCode=0 Aug 04 03:19:44.832580 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.832263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" event={"ID":"43031ab5-11a6-4b69-b581-2813b4db045f","Type":"ContainerDied","Data":"3dda88e5691cbaadce2f96abe96ddc097d19835cd1c513c906e8fc18ef8f6396"} Aug 04 03:19:44.834125 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.834108 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" Aug 04 03:19:44.834188 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.834153 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n" event={"ID":"36ad4170-0edc-490f-beac-3c807ea2da07","Type":"ContainerDied","Data":"a4002730d3b0807fac8372232f374bd5b4511558d58b5660873cc16338908d3d"} Aug 04 03:19:44.834238 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.834203 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4002730d3b0807fac8372232f374bd5b4511558d58b5660873cc16338908d3d" Aug 04 03:19:44.993481 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.993461 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:44.995529 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:44.995511 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:45.042556 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.042527 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-util\") pod \"e724473a-29da-4b06-aeb5-a8929046f137\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " Aug 04 03:19:45.042688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.042578 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42hk4\" (UniqueName: \"kubernetes.io/projected/e724473a-29da-4b06-aeb5-a8929046f137-kube-api-access-42hk4\") pod \"e724473a-29da-4b06-aeb5-a8929046f137\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " Aug 04 03:19:45.042688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.042639 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdgtc\" (UniqueName: \"kubernetes.io/projected/38e874d0-ae52-409a-9ac3-21243b5aa4d6-kube-api-access-qdgtc\") pod \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " Aug 04 03:19:45.042688 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.042671 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-bundle\") pod \"e724473a-29da-4b06-aeb5-a8929046f137\" (UID: \"e724473a-29da-4b06-aeb5-a8929046f137\") " Aug 04 03:19:45.042835 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.042696 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-util\") pod \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " Aug 04 03:19:45.042835 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.042763 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-bundle\") pod \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\" (UID: \"38e874d0-ae52-409a-9ac3-21243b5aa4d6\") " Aug 04 03:19:45.043571 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.043535 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-bundle" (OuterVolumeSpecName: "bundle") pod "e724473a-29da-4b06-aeb5-a8929046f137" (UID: "e724473a-29da-4b06-aeb5-a8929046f137"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:45.043669 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.043623 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-bundle" (OuterVolumeSpecName: "bundle") pod "38e874d0-ae52-409a-9ac3-21243b5aa4d6" (UID: "38e874d0-ae52-409a-9ac3-21243b5aa4d6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:45.045308 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.045280 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e874d0-ae52-409a-9ac3-21243b5aa4d6-kube-api-access-qdgtc" (OuterVolumeSpecName: "kube-api-access-qdgtc") pod "38e874d0-ae52-409a-9ac3-21243b5aa4d6" (UID: "38e874d0-ae52-409a-9ac3-21243b5aa4d6"). InnerVolumeSpecName "kube-api-access-qdgtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:19:45.045410 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.045385 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e724473a-29da-4b06-aeb5-a8929046f137-kube-api-access-42hk4" (OuterVolumeSpecName: "kube-api-access-42hk4") pod "e724473a-29da-4b06-aeb5-a8929046f137" (UID: "e724473a-29da-4b06-aeb5-a8929046f137"). InnerVolumeSpecName "kube-api-access-42hk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:19:45.049856 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.049823 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-util" (OuterVolumeSpecName: "util") pod "38e874d0-ae52-409a-9ac3-21243b5aa4d6" (UID: "38e874d0-ae52-409a-9ac3-21243b5aa4d6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:45.051749 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.051725 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-util" (OuterVolumeSpecName: "util") pod "e724473a-29da-4b06-aeb5-a8929046f137" (UID: "e724473a-29da-4b06-aeb5-a8929046f137"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:45.143988 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.143922 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-util\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:45.143988 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.143945 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hk4\" (UniqueName: \"kubernetes.io/projected/e724473a-29da-4b06-aeb5-a8929046f137-kube-api-access-42hk4\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:45.143988 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.143956 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdgtc\" (UniqueName: \"kubernetes.io/projected/38e874d0-ae52-409a-9ac3-21243b5aa4d6-kube-api-access-qdgtc\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:45.143988 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.143965 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e724473a-29da-4b06-aeb5-a8929046f137-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:45.143988 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.143973 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-util\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:45.143988 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.143981 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38e874d0-ae52-409a-9ac3-21243b5aa4d6-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:45.841187 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.841156 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" Aug 04 03:19:45.841643 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.841186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd" event={"ID":"38e874d0-ae52-409a-9ac3-21243b5aa4d6","Type":"ContainerDied","Data":"f8700858650339228f23625a9d59adba3820b86a59b9a51085c222b383e30d4c"} Aug 04 03:19:45.841643 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.841239 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8700858650339228f23625a9d59adba3820b86a59b9a51085c222b383e30d4c" Aug 04 03:19:45.842966 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.842935 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" event={"ID":"e724473a-29da-4b06-aeb5-a8929046f137","Type":"ContainerDied","Data":"e860cb9fb19bf0a65d0ee0054162c1f93f2607f47ee6e63681a8dca93acd8638"} Aug 04 03:19:45.843085 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.843022 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e860cb9fb19bf0a65d0ee0054162c1f93f2607f47ee6e63681a8dca93acd8638" Aug 04 03:19:45.843085 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.842950 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv" Aug 04 03:19:45.971341 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:45.971317 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:46.051347 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.051317 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-util\") pod \"43031ab5-11a6-4b69-b581-2813b4db045f\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " Aug 04 03:19:46.051503 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.051391 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7j9n\" (UniqueName: \"kubernetes.io/projected/43031ab5-11a6-4b69-b581-2813b4db045f-kube-api-access-g7j9n\") pod \"43031ab5-11a6-4b69-b581-2813b4db045f\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " Aug 04 03:19:46.051503 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.051415 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-bundle\") pod \"43031ab5-11a6-4b69-b581-2813b4db045f\" (UID: \"43031ab5-11a6-4b69-b581-2813b4db045f\") " Aug 04 03:19:46.051945 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.051911 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-bundle" (OuterVolumeSpecName: "bundle") pod "43031ab5-11a6-4b69-b581-2813b4db045f" (UID: "43031ab5-11a6-4b69-b581-2813b4db045f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:46.053449 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.053424 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43031ab5-11a6-4b69-b581-2813b4db045f-kube-api-access-g7j9n" (OuterVolumeSpecName: "kube-api-access-g7j9n") pod "43031ab5-11a6-4b69-b581-2813b4db045f" (UID: "43031ab5-11a6-4b69-b581-2813b4db045f"). InnerVolumeSpecName "kube-api-access-g7j9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 03:19:46.056526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.056508 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-util" (OuterVolumeSpecName: "util") pod "43031ab5-11a6-4b69-b581-2813b4db045f" (UID: "43031ab5-11a6-4b69-b581-2813b4db045f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 03:19:46.152856 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.152823 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-util\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:46.152856 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.152854 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7j9n\" (UniqueName: \"kubernetes.io/projected/43031ab5-11a6-4b69-b581-2813b4db045f-kube-api-access-g7j9n\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:46.153048 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.152865 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43031ab5-11a6-4b69-b581-2813b4db045f-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:19:46.850949 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.850924 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" Aug 04 03:19:46.850949 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.850929 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b" event={"ID":"43031ab5-11a6-4b69-b581-2813b4db045f","Type":"ContainerDied","Data":"2c9c6216a58f70867e0c602b4173138e8d468049b06ac23acf53302dabbcaa1d"} Aug 04 03:19:46.851352 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:19:46.850972 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c9c6216a58f70867e0c602b4173138e8d468049b06ac23acf53302dabbcaa1d" Aug 04 03:20:52.997182 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997146 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997460 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ad4170-0edc-490f-beac-3c807ea2da07" containerName="extract" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997470 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ad4170-0edc-490f-beac-3c807ea2da07" containerName="extract" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997479 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997485 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997490 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerName="extract" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997495 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerName="extract" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997504 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ad4170-0edc-490f-beac-3c807ea2da07" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997509 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ad4170-0edc-490f-beac-3c807ea2da07" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997519 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43031ab5-11a6-4b69-b581-2813b4db045f" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997524 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="43031ab5-11a6-4b69-b581-2813b4db045f" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997532 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43031ab5-11a6-4b69-b581-2813b4db045f" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997537 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="43031ab5-11a6-4b69-b581-2813b4db045f" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997547 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ad4170-0edc-490f-beac-3c807ea2da07" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997552 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ad4170-0edc-490f-beac-3c807ea2da07" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997558 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e724473a-29da-4b06-aeb5-a8929046f137" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997562 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e724473a-29da-4b06-aeb5-a8929046f137" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997568 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997573 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerName="util" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997578 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e724473a-29da-4b06-aeb5-a8929046f137" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997583 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e724473a-29da-4b06-aeb5-a8929046f137" containerName="pull" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997589 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43031ab5-11a6-4b69-b581-2813b4db045f" containerName="extract" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997594 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="43031ab5-11a6-4b69-b581-2813b4db045f" containerName="extract" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997607 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e724473a-29da-4b06-aeb5-a8929046f137" containerName="extract" Aug 04 03:20:52.997648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997612 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e724473a-29da-4b06-aeb5-a8929046f137" containerName="extract" Aug 04 03:20:52.998377 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997666 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="43031ab5-11a6-4b69-b581-2813b4db045f" containerName="extract" Aug 04 03:20:52.998377 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997673 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e724473a-29da-4b06-aeb5-a8929046f137" containerName="extract" Aug 04 03:20:52.998377 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997680 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e874d0-ae52-409a-9ac3-21243b5aa4d6" containerName="extract" Aug 04 03:20:52.998377 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:52.997687 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ad4170-0edc-490f-beac-3c807ea2da07" containerName="extract" Aug 04 03:20:53.000706 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.000690 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.003441 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.003418 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vjhhf\"" Aug 04 03:20:53.003441 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.003428 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 03:20:53.004527 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.004505 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 03:20:53.004614 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.004509 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 03:20:53.009286 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.009266 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:20:53.021836 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.021810 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:20:53.124768 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.124731 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1-config-file\") pod \"limitador-limitador-54d6d988d5-gtnfh\" (UID: \"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.124768 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.124771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8wf\" (UniqueName: \"kubernetes.io/projected/58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1-kube-api-access-sc8wf\") pod \"limitador-limitador-54d6d988d5-gtnfh\" (UID: \"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.225363 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.225323 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1-config-file\") pod \"limitador-limitador-54d6d988d5-gtnfh\" (UID: \"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.225363 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.225367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8wf\" (UniqueName: \"kubernetes.io/projected/58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1-kube-api-access-sc8wf\") pod \"limitador-limitador-54d6d988d5-gtnfh\" (UID: \"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.225978 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.225957 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1-config-file\") pod \"limitador-limitador-54d6d988d5-gtnfh\" (UID: \"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.235523 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.235489 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8wf\" (UniqueName: \"kubernetes.io/projected/58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1-kube-api-access-sc8wf\") pod \"limitador-limitador-54d6d988d5-gtnfh\" (UID: \"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.312115 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.312029 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:53.447967 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:53.447940 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:20:53.449824 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:20:53.449799 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bc4f81_2b90_4a4d_bcb6_9910aeff3dc1.slice/crio-d92d4e65d6931c6ea94b4ef08c6a4922735f513149712673c714b0ab68e4b559 WatchSource:0}: Error finding container d92d4e65d6931c6ea94b4ef08c6a4922735f513149712673c714b0ab68e4b559: Status 404 returned error can't find the container with id d92d4e65d6931c6ea94b4ef08c6a4922735f513149712673c714b0ab68e4b559 Aug 04 03:20:54.191992 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:54.191934 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" event={"ID":"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1","Type":"ContainerStarted","Data":"d92d4e65d6931c6ea94b4ef08c6a4922735f513149712673c714b0ab68e4b559"} Aug 04 03:20:56.204021 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:56.203980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" event={"ID":"58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1","Type":"ContainerStarted","Data":"f5500cb48bee85cb15ab76f0bd2c2270649f84821cd34ae05613b089ed62b859"} Aug 04 03:20:56.204411 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:56.204147 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:20:56.228865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:20:56.228814 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" podStartSLOduration=1.569051066 podStartE2EDuration="4.228799568s" totalImagesPullingTime="2.659748502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:20:56.2276846 +0000 UTC m=+679.348753962" watchObservedRunningTime="2026-08-04 03:20:56.228799568 +0000 UTC m=+679.349868910" Aug 04 03:21:07.210500 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:21:07.210467 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gtnfh" Aug 04 03:22:05.082957 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:22:05.082918 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:22:38.092697 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:22:38.092663 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:22:57.683184 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:22:57.683147 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:23:10.181268 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:10.181234 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:23:24.073392 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.073359 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c"] Aug 04 03:23:24.076852 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.076827 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.079403 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.079375 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 03:23:24.080733 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.080713 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 03:23:24.080862 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.080741 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 03:23:24.080862 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.080794 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jmw7d\"" Aug 04 03:23:24.087723 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.087698 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c"] Aug 04 03:23:24.110687 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.110666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.110805 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.110694 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.110805 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.110713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.110805 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.110733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.110805 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.110759 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8n6g\" (UniqueName: \"kubernetes.io/projected/4d118ba8-88b5-4166-b99d-26860cc0c4c8-kube-api-access-m8n6g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.110959 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.110841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d118ba8-88b5-4166-b99d-26860cc0c4c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211277 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d118ba8-88b5-4166-b99d-26860cc0c4c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211303 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211633 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211512 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8n6g\" (UniqueName: \"kubernetes.io/projected/4d118ba8-88b5-4166-b99d-26860cc0c4c8-kube-api-access-m8n6g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211731 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211707 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211792 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211749 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.211792 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.211772 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.213567 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.213547 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d118ba8-88b5-4166-b99d-26860cc0c4c8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.213761 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.213743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d118ba8-88b5-4166-b99d-26860cc0c4c8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.219118 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.219097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8n6g\" (UniqueName: \"kubernetes.io/projected/4d118ba8-88b5-4166-b99d-26860cc0c4c8-kube-api-access-m8n6g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c\" (UID: \"4d118ba8-88b5-4166-b99d-26860cc0c4c8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.389043 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.388965 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:24.518701 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.518626 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c"] Aug 04 03:23:24.522113 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:23:24.522083 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d118ba8_88b5_4166_b99d_26860cc0c4c8.slice/crio-cbb21bc3d435a584af9bb0b73760fa5d6dda2c2c3b442c62b74544d3427deb34 WatchSource:0}: Error finding container cbb21bc3d435a584af9bb0b73760fa5d6dda2c2c3b442c62b74544d3427deb34: Status 404 returned error can't find the container with id cbb21bc3d435a584af9bb0b73760fa5d6dda2c2c3b442c62b74544d3427deb34 Aug 04 03:23:24.931871 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:24.931834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" event={"ID":"4d118ba8-88b5-4166-b99d-26860cc0c4c8","Type":"ContainerStarted","Data":"cbb21bc3d435a584af9bb0b73760fa5d6dda2c2c3b442c62b74544d3427deb34"} Aug 04 03:23:25.588009 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:25.587980 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:23:25.938354 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:25.938321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" event={"ID":"4d118ba8-88b5-4166-b99d-26860cc0c4c8","Type":"ContainerStarted","Data":"0a7112a11c78b3ae4327d866d3e70ffedf24962c3148736bd490beaad505405e"} Aug 04 03:23:25.958888 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:25.958837 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" podStartSLOduration=0.990460065 podStartE2EDuration="1.958821131s" totalImagesPullingTime="968.361066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:23:25.95684321 +0000 UTC m=+829.077912570" watchObservedRunningTime="2026-08-04 03:23:25.958821131 +0000 UTC m=+829.079890473" Aug 04 03:23:34.389912 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:34.389834 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:34.389912 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:34.389928 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:34.402198 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:34.402166 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:34.996759 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:34.996734 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c" Aug 04 03:23:47.783230 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.783193 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k"] Aug 04 03:23:47.786954 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.786932 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:47.789307 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.789282 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 04 03:23:47.796106 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.796080 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k"] Aug 04 03:23:47.914882 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.914842 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f065657-685a-4be9-81e1-7562b6095ef6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:47.915102 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.914917 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kmv\" (UniqueName: \"kubernetes.io/projected/2f065657-685a-4be9-81e1-7562b6095ef6-kube-api-access-t8kmv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:47.915102 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.914946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:47.915102 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.914986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:47.915102 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.915009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:47.915256 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:47.915094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016188 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016149 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016378 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016210 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f065657-685a-4be9-81e1-7562b6095ef6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016378 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kmv\" (UniqueName: \"kubernetes.io/projected/2f065657-685a-4be9-81e1-7562b6095ef6-kube-api-access-t8kmv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016378 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016378 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016378 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016680 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016661 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016764 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.016803 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.016760 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.018494 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.018475 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f065657-685a-4be9-81e1-7562b6095ef6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.018746 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.018729 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f065657-685a-4be9-81e1-7562b6095ef6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.024298 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.024276 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kmv\" (UniqueName: \"kubernetes.io/projected/2f065657-685a-4be9-81e1-7562b6095ef6-kube-api-access-t8kmv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-sct9k\" (UID: \"2f065657-685a-4be9-81e1-7562b6095ef6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.098823 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.098744 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:48.225812 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:48.225787 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k"] Aug 04 03:23:48.227656 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:23:48.227624 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f065657_685a_4be9_81e1_7562b6095ef6.slice/crio-67be903b8374064584309a95f1ce912b059302d11bc53d2446ef64600466a82b WatchSource:0}: Error finding container 67be903b8374064584309a95f1ce912b059302d11bc53d2446ef64600466a82b: Status 404 returned error can't find the container with id 67be903b8374064584309a95f1ce912b059302d11bc53d2446ef64600466a82b Aug 04 03:23:49.058532 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:49.058491 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" event={"ID":"2f065657-685a-4be9-81e1-7562b6095ef6","Type":"ContainerStarted","Data":"a0962326f34dd010fe92d618136aae5af574b671a95b5eed3f98b5c6290798df"} Aug 04 03:23:49.058931 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:49.058551 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" event={"ID":"2f065657-685a-4be9-81e1-7562b6095ef6","Type":"ContainerStarted","Data":"67be903b8374064584309a95f1ce912b059302d11bc53d2446ef64600466a82b"} Aug 04 03:23:49.078913 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:49.077699 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" podStartSLOduration=1.919357035 podStartE2EDuration="2.077680737s" totalImagesPullingTime="158.323702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:23:49.075392685 +0000 UTC m=+852.196462048" watchObservedRunningTime="2026-08-04 03:23:49.077680737 +0000 UTC m=+852.198750080" Aug 04 03:23:50.412005 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:50.411971 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:23:58.099166 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:58.099139 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:58.099566 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:58.099236 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:58.111332 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:58.111313 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:23:59.118599 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:23:59.118573 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-sct9k" Aug 04 03:24:00.697268 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.697228 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7"] Aug 04 03:24:00.700785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.700759 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.703227 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.703210 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 03:24:00.712545 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.712524 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7"] Aug 04 03:24:00.826301 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.826271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.826301 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.826300 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt26t\" (UniqueName: \"kubernetes.io/projected/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-kube-api-access-pt26t\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.826526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.826328 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.826526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.826430 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.826526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.826476 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.826526 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.826501 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927064 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927028 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927064 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt26t\" (UniqueName: \"kubernetes.io/projected/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-kube-api-access-pt26t\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927257 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927106 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927257 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927257 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927405 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927498 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927476 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927586 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927566 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.927648 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.927614 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.929435 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.929411 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.929625 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.929611 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:00.935000 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:00.934983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt26t\" (UniqueName: \"kubernetes.io/projected/e8f1741b-5467-4df3-89f0-adbd1cbdfa17-kube-api-access-pt26t\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7\" (UID: \"e8f1741b-5467-4df3-89f0-adbd1cbdfa17\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:01.010126 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:01.010044 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:01.142695 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:01.142670 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7"] Aug 04 03:24:01.144018 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:24:01.143990 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f1741b_5467_4df3_89f0_adbd1cbdfa17.slice/crio-881c9322449f2a1374799b4b73838eca6934645ea4d48a316c36d6c54d513bd1 WatchSource:0}: Error finding container 881c9322449f2a1374799b4b73838eca6934645ea4d48a316c36d6c54d513bd1: Status 404 returned error can't find the container with id 881c9322449f2a1374799b4b73838eca6934645ea4d48a316c36d6c54d513bd1 Aug 04 03:24:02.126852 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:02.126818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" event={"ID":"e8f1741b-5467-4df3-89f0-adbd1cbdfa17","Type":"ContainerStarted","Data":"b0c7f7407573c82cef02edf4fa640f07fea711dbf35c8ec44b8c123d92d8b91d"} Aug 04 03:24:02.126852 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:02.126856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" event={"ID":"e8f1741b-5467-4df3-89f0-adbd1cbdfa17","Type":"ContainerStarted","Data":"881c9322449f2a1374799b4b73838eca6934645ea4d48a316c36d6c54d513bd1"} Aug 04 03:24:02.147639 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:02.147586 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" podStartSLOduration=1.985145913 podStartE2EDuration="2.147570522s" totalImagesPullingTime="162.424609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 03:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 03:24:02.144439934 +0000 UTC m=+865.265509296" watchObservedRunningTime="2026-08-04 03:24:02.147570522 +0000 UTC m=+865.268639864" Aug 04 03:24:03.983843 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:03.983808 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:24:11.011193 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:11.011162 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:11.011577 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:11.011206 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:11.023742 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:11.023710 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:11.186225 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:11.186189 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7" Aug 04 03:24:39.676739 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:39.676706 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:24:39.680462 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:24:39.680444 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:25:56.488909 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:25:56.488869 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:26:07.586047 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:26:07.586017 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:26:17.919767 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:26:17.919731 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:27:58.789103 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:27:58.789068 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:28:11.479049 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:28:11.479018 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:29:22.878615 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:29:22.878580 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:29:32.182332 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:29:32.182295 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:29:39.710614 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:29:39.710505 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:29:39.715542 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:29:39.714607 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:29:47.292621 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:29:47.292589 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:29:56.177281 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:29:56.177245 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:30:07.189242 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:30:07.189210 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:30:17.579366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:30:17.579331 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:30:30.287563 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:30:30.287369 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:30:40.186660 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:30:40.186626 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:31:01.582910 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:31:01.582814 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:31:24.084098 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:31:24.084062 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:31:45.584656 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:31:45.584621 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:31:56.187465 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:31:56.187427 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:32:12.983114 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:32:12.983029 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:32:31.781386 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:32:31.781328 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:32:47.593093 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:32:47.593058 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:32:57.491849 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:32:57.491809 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:33:07.287427 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:33:07.287386 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:33:19.383771 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:33:19.383738 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:33:29.381524 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:33:29.381452 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:33:40.494885 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:33:40.494851 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:34:01.785615 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:34:01.785577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:34:39.745586 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:34:39.745472 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:34:39.751388 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:34:39.751366 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:36:29.079299 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:36:29.079217 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:36:39.681121 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:36:39.681085 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:36:58.894427 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:36:58.894383 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:37:30.489847 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:37:30.489800 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:37:39.481687 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:37:39.481651 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:37:49.480279 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:37:49.480245 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:37:59.184092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:37:59.184013 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:38:10.885910 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:38:10.885866 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:38:21.975701 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:38:21.975661 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:38:33.881374 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:38:33.881336 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:38:45.375732 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:38:45.375699 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:38:53.190335 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:38:53.190301 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:39:11.790665 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:39:11.790630 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:39:22.688529 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:39:22.688494 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:39:37.680181 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:39:37.680133 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:39:39.780929 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:39:39.780791 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:39:39.787948 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:39:39.787929 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:40:00.334715 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:40:00.334680 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:40:14.090734 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:40:14.090697 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:40:21.886040 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:40:21.886000 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:40:30.586825 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:40:30.586790 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:40:37.180603 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:40:37.180559 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:40:51.588673 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:40:51.588638 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:41:00.788522 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:41:00.788433 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:41:11.482712 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:41:11.482675 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:41:20.880997 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:41:20.880959 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:41:30.781303 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:41:30.781263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:41:44.092128 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:41:44.092090 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:41:57.875283 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:41:57.875247 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:42:07.382092 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:42:07.382058 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:42:16.591001 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:42:16.590962 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:42:28.780121 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:42:28.780080 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:42:40.088946 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:42:40.088879 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:42:55.483024 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:42:55.482990 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:43:07.885468 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:43:07.881491 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:43:22.788685 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:43:22.788652 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:43:48.191721 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:43:48.191686 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:43:57.583703 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:43:57.583661 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:44:10.588184 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:44:10.588103 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:44:25.090585 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:44:25.090551 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:44:34.785667 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:44:34.785630 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:44:39.814979 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:44:39.814846 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:44:39.822812 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:44:39.822792 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:44:44.481735 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:44:44.481703 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:44:55.188772 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:44:55.188728 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:45:23.184582 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:45:23.184544 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:45:34.884980 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:45:34.884946 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:45:57.590555 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:45:57.590524 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:46:09.988449 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:46:09.988409 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:46:32.781948 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:46:32.781911 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:46:44.989566 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:46:44.989535 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:47:06.680855 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:47:06.680778 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:47:15.984651 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:47:15.984613 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:47:30.785220 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:47:30.785184 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:48:59.681546 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:48:59.681509 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:49:09.675653 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:49:09.675619 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:49:19.387914 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:49:19.387864 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:49:39.848434 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:49:39.848334 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:49:39.864199 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:49:39.864175 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:50:00.281615 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:50:00.281531 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:50:35.490723 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:50:35.490686 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:52:21.284027 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:52:21.283992 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:52:32.080420 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:52:32.080385 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:52:52.180307 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:52:52.180275 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:53:05.898366 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:53:05.898290 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:53:29.583837 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:53:29.583806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:53:37.884472 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:53:37.884431 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:54:30.694067 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:54:30.693974 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:54:39.895103 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:54:39.894992 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:54:39.904619 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:54:39.904599 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:54:43.786399 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:54:43.786366 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:54:51.781195 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:54:51.781160 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:55:42.794142 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:55:42.794104 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:56:13.493424 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:56:13.493343 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:56:58.394169 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:56:58.394133 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:57:12.288537 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:57:12.288498 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:57:28.889934 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:57:28.889826 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:58:07.929730 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:07.929693 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj"] Aug 04 03:58:07.933102 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:07.933086 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" Aug 04 03:58:07.935640 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:07.935620 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 03:58:07.935722 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:07.935706 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 03:58:07.939865 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:07.939844 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj"] Aug 04 03:58:08.025998 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:08.025959 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f93e809-a099-4fa6-8378-84d5f0d86a02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj\" (UID: \"3f93e809-a099-4fa6-8378-84d5f0d86a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" Aug 04 03:58:08.126622 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:08.126595 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f93e809-a099-4fa6-8378-84d5f0d86a02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj\" (UID: \"3f93e809-a099-4fa6-8378-84d5f0d86a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" Aug 04 03:58:08.127192 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:08.127173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f93e809-a099-4fa6-8378-84d5f0d86a02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj\" (UID: \"3f93e809-a099-4fa6-8378-84d5f0d86a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" Aug 04 03:58:08.245146 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:08.245050 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" Aug 04 03:58:08.373516 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:08.373487 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj"] Aug 04 03:58:08.375244 ip-10-0-130-28 kubenswrapper[2568]: W0804 03:58:08.375211 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f93e809_a099_4fa6_8378_84d5f0d86a02.slice/crio-2929517717f494cf84e910ca8bb2a0cacee9802e080d8626d42fc7aab6afbf06 WatchSource:0}: Error finding container 2929517717f494cf84e910ca8bb2a0cacee9802e080d8626d42fc7aab6afbf06: Status 404 returned error can't find the container with id 2929517717f494cf84e910ca8bb2a0cacee9802e080d8626d42fc7aab6afbf06 Aug 04 03:58:08.377348 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:08.377332 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 03:58:08.734550 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:08.734515 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" event={"ID":"3f93e809-a099-4fa6-8378-84d5f0d86a02","Type":"ContainerStarted","Data":"2929517717f494cf84e910ca8bb2a0cacee9802e080d8626d42fc7aab6afbf06"} Aug 04 03:58:11.587539 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:11.587436 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:58:11.753186 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:11.753149 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f93e809-a099-4fa6-8378-84d5f0d86a02" containerID="05a25f3c6bf7e810aab3c90347fb8c08d72701beb5b5f73a1694d4f98a349e2d" exitCode=0 Aug 04 03:58:11.753359 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:11.753244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" event={"ID":"3f93e809-a099-4fa6-8378-84d5f0d86a02","Type":"ContainerDied","Data":"05a25f3c6bf7e810aab3c90347fb8c08d72701beb5b5f73a1694d4f98a349e2d"} Aug 04 03:58:12.892365 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:12.892342 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" Aug 04 03:58:12.967239 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:12.967199 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f93e809-a099-4fa6-8378-84d5f0d86a02-maas-api-service-ca\") pod \"3f93e809-a099-4fa6-8378-84d5f0d86a02\" (UID: \"3f93e809-a099-4fa6-8378-84d5f0d86a02\") " Aug 04 03:58:12.967475 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:12.967452 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f93e809-a099-4fa6-8378-84d5f0d86a02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f93e809-a099-4fa6-8378-84d5f0d86a02" (UID: "3f93e809-a099-4fa6-8378-84d5f0d86a02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 03:58:13.068288 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:13.068201 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f93e809-a099-4fa6-8378-84d5f0d86a02-maas-api-service-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 03:58:13.765021 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:13.764984 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" event={"ID":"3f93e809-a099-4fa6-8378-84d5f0d86a02","Type":"ContainerDied","Data":"2929517717f494cf84e910ca8bb2a0cacee9802e080d8626d42fc7aab6afbf06"} Aug 04 03:58:13.765021 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:13.765017 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj" Aug 04 03:58:13.765259 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:13.765041 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2929517717f494cf84e910ca8bb2a0cacee9802e080d8626d42fc7aab6afbf06" Aug 04 03:58:18.695044 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:18.695006 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj"] Aug 04 03:58:18.697993 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:18.697969 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a60de-q4hfj"] Aug 04 03:58:19.372889 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:19.372855 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f93e809-a099-4fa6-8378-84d5f0d86a02" path="/var/lib/kubelet/pods/3f93e809-a099-4fa6-8378-84d5f0d86a02/volumes" Aug 04 03:58:24.788785 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:24.788751 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:58:38.683960 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:38.683924 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:58:53.883725 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:58:53.883686 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:59:21.687108 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:59:21.687030 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 03:59:39.929160 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:59:39.929128 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 03:59:39.940297 ip-10-0-130-28 kubenswrapper[2568]: I0804 03:59:39.940275 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:00:00.151783 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.151742 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn"] Aug 04 04:00:00.152272 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.152132 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f93e809-a099-4fa6-8378-84d5f0d86a02" containerName="revoke-keys" Aug 04 04:00:00.152272 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.152146 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f93e809-a099-4fa6-8378-84d5f0d86a02" containerName="revoke-keys" Aug 04 04:00:00.152272 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.152215 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f93e809-a099-4fa6-8378-84d5f0d86a02" containerName="revoke-keys" Aug 04 04:00:00.155266 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.155251 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" Aug 04 04:00:00.158064 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.158044 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 04:00:00.165453 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.165429 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn"] Aug 04 04:00:00.166977 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.166960 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" Aug 04 04:00:00.508095 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:00.508072 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn"] Aug 04 04:00:00.510149 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:00:00.510114 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ea7de5_f561_41cb_ae99_ba617bde7bac.slice/crio-1863ff8ae9f723450be9f0ce3b819e39547250ca41300461d8967156bf042da0 WatchSource:0}: Error finding container 1863ff8ae9f723450be9f0ce3b819e39547250ca41300461d8967156bf042da0: Status 404 returned error can't find the container with id 1863ff8ae9f723450be9f0ce3b819e39547250ca41300461d8967156bf042da0 Aug 04 04:00:01.342224 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:01.342194 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" event={"ID":"e6ea7de5-f561-41cb-ae99-ba617bde7bac","Type":"ContainerStarted","Data":"789a3e510f6806e9c3fae1b289f14e9fd713c92473f6f5d57081e03ac5b3541b"} Aug 04 04:00:01.342634 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:01.342242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" event={"ID":"e6ea7de5-f561-41cb-ae99-ba617bde7bac","Type":"ContainerStarted","Data":"1863ff8ae9f723450be9f0ce3b819e39547250ca41300461d8967156bf042da0"} Aug 04 04:00:01.358405 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:01.358361 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" podStartSLOduration=1.35834733 podStartE2EDuration="1.35834733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:00:01.356704828 +0000 UTC m=+3024.477774170" watchObservedRunningTime="2026-08-04 04:00:01.35834733 +0000 UTC m=+3024.479416672" Aug 04 04:00:02.349465 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:02.349427 2568 generic.go:356] "Generic (PLEG): container finished" podID="e6ea7de5-f561-41cb-ae99-ba617bde7bac" containerID="789a3e510f6806e9c3fae1b289f14e9fd713c92473f6f5d57081e03ac5b3541b" exitCode=0 Aug 04 04:00:02.349843 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:02.349508 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" event={"ID":"e6ea7de5-f561-41cb-ae99-ba617bde7bac","Type":"ContainerDied","Data":"789a3e510f6806e9c3fae1b289f14e9fd713c92473f6f5d57081e03ac5b3541b"} Aug 04 04:00:03.480999 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:03.480975 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" Aug 04 04:00:04.363752 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:04.363723 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" Aug 04 04:00:04.363752 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:04.363733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn" event={"ID":"e6ea7de5-f561-41cb-ae99-ba617bde7bac","Type":"ContainerDied","Data":"1863ff8ae9f723450be9f0ce3b819e39547250ca41300461d8967156bf042da0"} Aug 04 04:00:04.363982 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:04.363784 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1863ff8ae9f723450be9f0ce3b819e39547250ca41300461d8967156bf042da0" Aug 04 04:00:05.390625 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:05.390235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:00:21.664667 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:21.664628 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn"] Aug 04 04:00:21.667010 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:21.666988 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-523226-29763600-4htmn"] Aug 04 04:00:23.373657 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:23.373620 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6ea7de5-f561-41cb-ae99-ba617bde7bac" path="/var/lib/kubelet/pods/e6ea7de5-f561-41cb-ae99-ba617bde7bac/volumes" Aug 04 04:00:26.719887 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:26.719847 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:00:37.897806 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:00:37.897771 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:01:08.893501 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:01:08.893468 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:01:35.514858 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:01:35.514822 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:02:06.215035 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:02:06.214944 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:02:43.591620 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:02:43.591576 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:03:08.693845 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:03:08.693810 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:03:28.091610 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:03:28.091570 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:03:55.106508 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:03:55.106472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:04:08.387909 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:08.387864 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:04:19.699481 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:19.699449 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:04:24.142931 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.142871 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hxbwb"] Aug 04 04:04:24.143312 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.143281 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6ea7de5-f561-41cb-ae99-ba617bde7bac" containerName="cleanup" Aug 04 04:04:24.143312 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.143293 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6ea7de5-f561-41cb-ae99-ba617bde7bac" containerName="cleanup" Aug 04 04:04:24.143387 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.143357 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6ea7de5-f561-41cb-ae99-ba617bde7bac" containerName="cleanup" Aug 04 04:04:24.146329 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.146311 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:24.149222 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.149202 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-78vtm\"" Aug 04 04:04:24.165213 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.165185 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hxbwb"] Aug 04 04:04:24.268515 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.268479 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mf48\" (UniqueName: \"kubernetes.io/projected/e3722c01-47a5-440e-827e-5bc25ff5f1fa-kube-api-access-9mf48\") pod \"kuadrant-operator-catalog-hxbwb\" (UID: \"e3722c01-47a5-440e-827e-5bc25ff5f1fa\") " pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:24.369712 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.369671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mf48\" (UniqueName: \"kubernetes.io/projected/e3722c01-47a5-440e-827e-5bc25ff5f1fa-kube-api-access-9mf48\") pod \"kuadrant-operator-catalog-hxbwb\" (UID: \"e3722c01-47a5-440e-827e-5bc25ff5f1fa\") " pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:24.379201 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.379173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mf48\" (UniqueName: \"kubernetes.io/projected/e3722c01-47a5-440e-827e-5bc25ff5f1fa-kube-api-access-9mf48\") pod \"kuadrant-operator-catalog-hxbwb\" (UID: \"e3722c01-47a5-440e-827e-5bc25ff5f1fa\") " pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:24.455431 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.455399 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:24.590803 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.590776 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hxbwb"] Aug 04 04:04:24.592642 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:04:24.592617 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3722c01_47a5_440e_827e_5bc25ff5f1fa.slice/crio-99d870aa592491e7456254ade390969a241941e9ad1c24fc232221b9e846127e WatchSource:0}: Error finding container 99d870aa592491e7456254ade390969a241941e9ad1c24fc232221b9e846127e: Status 404 returned error can't find the container with id 99d870aa592491e7456254ade390969a241941e9ad1c24fc232221b9e846127e Aug 04 04:04:24.594078 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.594054 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 04:04:24.709021 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:24.708938 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" event={"ID":"e3722c01-47a5-440e-827e-5bc25ff5f1fa","Type":"ContainerStarted","Data":"99d870aa592491e7456254ade390969a241941e9ad1c24fc232221b9e846127e"} Aug 04 04:04:26.724237 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:26.724196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" event={"ID":"e3722c01-47a5-440e-827e-5bc25ff5f1fa","Type":"ContainerStarted","Data":"2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280"} Aug 04 04:04:26.743162 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:26.743105 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" podStartSLOduration=0.712935686 podStartE2EDuration="2.743086802s" totalImagesPullingTime="2.030151116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:04:26.74059449 +0000 UTC m=+3289.861663858" watchObservedRunningTime="2026-08-04 04:04:26.743086802 +0000 UTC m=+3289.864156143" Aug 04 04:04:34.455953 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:34.455831 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:34.455953 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:34.455929 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:34.477966 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:34.477932 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:34.788175 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:34.788103 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:34.828653 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:34.828621 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hxbwb"] Aug 04 04:04:36.776745 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:36.776701 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" podUID="e3722c01-47a5-440e-827e-5bc25ff5f1fa" containerName="registry-server" containerID="cri-o://2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280" gracePeriod=2 Aug 04 04:04:37.036328 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.036267 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:37.084632 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.084605 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mf48\" (UniqueName: \"kubernetes.io/projected/e3722c01-47a5-440e-827e-5bc25ff5f1fa-kube-api-access-9mf48\") pod \"e3722c01-47a5-440e-827e-5bc25ff5f1fa\" (UID: \"e3722c01-47a5-440e-827e-5bc25ff5f1fa\") " Aug 04 04:04:37.086672 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.086649 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3722c01-47a5-440e-827e-5bc25ff5f1fa-kube-api-access-9mf48" (OuterVolumeSpecName: "kube-api-access-9mf48") pod "e3722c01-47a5-440e-827e-5bc25ff5f1fa" (UID: "e3722c01-47a5-440e-827e-5bc25ff5f1fa"). InnerVolumeSpecName "kube-api-access-9mf48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 04:04:37.185528 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.185478 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mf48\" (UniqueName: \"kubernetes.io/projected/e3722c01-47a5-440e-827e-5bc25ff5f1fa-kube-api-access-9mf48\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:04:37.643459 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.643372 2568 scope.go:117] "RemoveContainer" containerID="05a25f3c6bf7e810aab3c90347fb8c08d72701beb5b5f73a1694d4f98a349e2d" Aug 04 04:04:37.788958 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.788927 2568 generic.go:356] "Generic (PLEG): container finished" podID="e3722c01-47a5-440e-827e-5bc25ff5f1fa" containerID="2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280" exitCode=0 Aug 04 04:04:37.789375 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.789023 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" Aug 04 04:04:37.789375 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.789051 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" event={"ID":"e3722c01-47a5-440e-827e-5bc25ff5f1fa","Type":"ContainerDied","Data":"2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280"} Aug 04 04:04:37.789375 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.789117 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hxbwb" event={"ID":"e3722c01-47a5-440e-827e-5bc25ff5f1fa","Type":"ContainerDied","Data":"99d870aa592491e7456254ade390969a241941e9ad1c24fc232221b9e846127e"} Aug 04 04:04:37.789375 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.789136 2568 scope.go:117] "RemoveContainer" containerID="2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280" Aug 04 04:04:37.798148 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.798132 2568 scope.go:117] "RemoveContainer" containerID="2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280" Aug 04 04:04:37.798430 ip-10-0-130-28 kubenswrapper[2568]: E0804 04:04:37.798413 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280\": container with ID starting with 2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280 not found: ID does not exist" containerID="2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280" Aug 04 04:04:37.798513 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.798439 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280"} err="failed to get container status \"2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280\": rpc error: code = NotFound desc = could not find container \"2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280\": container with ID starting with 2ab80e8a86d28be27b90ce2a4defc26a6eb90cddfc71753b12fa56e156ae7280 not found: ID does not exist" Aug 04 04:04:37.809966 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.809942 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hxbwb"] Aug 04 04:04:37.813326 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:37.813304 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hxbwb"] Aug 04 04:04:39.104039 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:39.104006 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:04:39.372846 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:39.372771 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3722c01-47a5-440e-827e-5bc25ff5f1fa" path="/var/lib/kubelet/pods/e3722c01-47a5-440e-827e-5bc25ff5f1fa/volumes" Aug 04 04:04:39.963651 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:39.963546 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:04:39.978747 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:39.975254 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:04:52.580778 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:04:52.580745 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:05:04.496072 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:05:04.495984 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:05:23.882620 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:05:23.882586 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:05:37.685192 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:05:37.685156 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:05:50.198356 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:05:50.198321 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:06:09.091561 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:06:09.091529 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:06:26.693021 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:06:26.692981 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:06:37.676447 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:06:37.676419 2568 scope.go:117] "RemoveContainer" containerID="789a3e510f6806e9c3fae1b289f14e9fd713c92473f6f5d57081e03ac5b3541b" Aug 04 04:06:53.893105 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:06:53.893069 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:07:09.306368 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.306329 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb"] Aug 04 04:07:09.306782 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.306679 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3722c01-47a5-440e-827e-5bc25ff5f1fa" containerName="registry-server" Aug 04 04:07:09.306782 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.306690 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3722c01-47a5-440e-827e-5bc25ff5f1fa" containerName="registry-server" Aug 04 04:07:09.306782 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.306760 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3722c01-47a5-440e-827e-5bc25ff5f1fa" containerName="registry-server" Aug 04 04:07:09.309803 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.309784 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.313720 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.313642 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-07c848\"/\"sub-test-model-07c848-kserve-self-signed-certs\"" Aug 04 04:07:09.313720 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.313666 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-07c848\"/\"kube-root-ca.crt\"" Aug 04 04:07:09.313720 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.313711 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-07c848\"/\"openshift-service-ca.crt\"" Aug 04 04:07:09.313961 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.313730 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-07c848\"/\"default-dockercfg-9t4ck\"" Aug 04 04:07:09.319114 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.319093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb"] Aug 04 04:07:09.404531 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.404498 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tls-certs\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.404531 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.404530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n758w\" (UniqueName: \"kubernetes.io/projected/0ee256a1-73bc-48e1-8912-e50ddd4320b1-kube-api-access-n758w\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.404775 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.404563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-dshm\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.404775 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.404582 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-home\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.404775 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.404689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tmp-dir\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.404775 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.404744 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-model-cache\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505237 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-model-cache\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505431 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tls-certs\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505431 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505281 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n758w\" (UniqueName: \"kubernetes.io/projected/0ee256a1-73bc-48e1-8912-e50ddd4320b1-kube-api-access-n758w\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505431 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-dshm\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505431 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505374 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-home\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505431 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tmp-dir\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505724 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505694 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-model-cache\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505777 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505740 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tmp-dir\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.505843 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.505819 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-home\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.507554 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.507538 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-dshm\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.507714 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.507697 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tls-certs\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.514462 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.514439 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n758w\" (UniqueName: \"kubernetes.io/projected/0ee256a1-73bc-48e1-8912-e50ddd4320b1-kube-api-access-n758w\") pod \"sub-test-model-07c848-kserve-6fc556964c-tt2sb\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.619981 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.619882 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:09.726478 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.726424 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:07:09.767586 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:09.767560 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb"] Aug 04 04:07:09.768862 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:07:09.768832 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ee256a1_73bc_48e1_8912_e50ddd4320b1.slice/crio-02da4591597b3d5b3dd7c5092b9f5d2d9d1372a9955e131781cdd08040659644 WatchSource:0}: Error finding container 02da4591597b3d5b3dd7c5092b9f5d2d9d1372a9955e131781cdd08040659644: Status 404 returned error can't find the container with id 02da4591597b3d5b3dd7c5092b9f5d2d9d1372a9955e131781cdd08040659644 Aug 04 04:07:10.583413 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:10.583372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" event={"ID":"0ee256a1-73bc-48e1-8912-e50ddd4320b1","Type":"ContainerStarted","Data":"75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a"} Aug 04 04:07:10.583413 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:10.583416 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" event={"ID":"0ee256a1-73bc-48e1-8912-e50ddd4320b1","Type":"ContainerStarted","Data":"02da4591597b3d5b3dd7c5092b9f5d2d9d1372a9955e131781cdd08040659644"} Aug 04 04:07:10.602839 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:10.602786 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" podStartSLOduration=1.6027700550000001 podStartE2EDuration="1.602770055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:07:10.600676682 +0000 UTC m=+3453.721746026" watchObservedRunningTime="2026-08-04 04:07:10.602770055 +0000 UTC m=+3453.723839397" Aug 04 04:07:19.286983 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:19.286945 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:07:19.620374 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:19.620275 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:19.620374 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:19.620322 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:19.633002 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:19.632976 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:19.644271 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:19.644247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:07:45.711304 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:07:45.711271 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:08:42.196744 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:08:42.196705 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:09:02.198606 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:02.198564 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:09:23.991739 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:23.991707 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:09:28.926236 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:28.926144 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb"] Aug 04 04:09:39.998538 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:39.998425 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:09:40.010863 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:40.010844 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:09:42.283223 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:42.283192 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:09:43.945123 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:43.945076 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" podUID="0ee256a1-73bc-48e1-8912-e50ddd4320b1" containerName="main" containerID="cri-o://75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a" gracePeriod=45 Aug 04 04:09:44.187412 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.187386 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:09:44.277225 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277134 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-home\") pod \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " Aug 04 04:09:44.277397 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277242 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-dshm\") pod \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " Aug 04 04:09:44.277397 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277281 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n758w\" (UniqueName: \"kubernetes.io/projected/0ee256a1-73bc-48e1-8912-e50ddd4320b1-kube-api-access-n758w\") pod \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " Aug 04 04:09:44.277397 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277340 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tls-certs\") pod \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " Aug 04 04:09:44.277397 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277389 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-model-cache\") pod \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " Aug 04 04:09:44.277625 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277384 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-home" (OuterVolumeSpecName: "home") pod "0ee256a1-73bc-48e1-8912-e50ddd4320b1" (UID: "0ee256a1-73bc-48e1-8912-e50ddd4320b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:09:44.277625 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277448 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tmp-dir\") pod \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\" (UID: \"0ee256a1-73bc-48e1-8912-e50ddd4320b1\") " Aug 04 04:09:44.277625 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277589 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ee256a1-73bc-48e1-8912-e50ddd4320b1" (UID: "0ee256a1-73bc-48e1-8912-e50ddd4320b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:09:44.277763 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277722 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ee256a1-73bc-48e1-8912-e50ddd4320b1" (UID: "0ee256a1-73bc-48e1-8912-e50ddd4320b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:09:44.277763 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277730 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:09:44.277875 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.277786 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:09:44.279468 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.279444 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-dshm" (OuterVolumeSpecName: "dshm") pod "0ee256a1-73bc-48e1-8912-e50ddd4320b1" (UID: "0ee256a1-73bc-48e1-8912-e50ddd4320b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:09:44.279564 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.279483 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee256a1-73bc-48e1-8912-e50ddd4320b1-kube-api-access-n758w" (OuterVolumeSpecName: "kube-api-access-n758w") pod "0ee256a1-73bc-48e1-8912-e50ddd4320b1" (UID: "0ee256a1-73bc-48e1-8912-e50ddd4320b1"). InnerVolumeSpecName "kube-api-access-n758w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 04:09:44.279564 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.279494 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ee256a1-73bc-48e1-8912-e50ddd4320b1" (UID: "0ee256a1-73bc-48e1-8912-e50ddd4320b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 04:09:44.379000 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.378968 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:09:44.379000 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.378994 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n758w\" (UniqueName: \"kubernetes.io/projected/0ee256a1-73bc-48e1-8912-e50ddd4320b1-kube-api-access-n758w\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:09:44.379000 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.379005 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:09:44.379233 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.379014 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ee256a1-73bc-48e1-8912-e50ddd4320b1-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:09:44.385491 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.385465 2568 generic.go:356] "Generic (PLEG): container finished" podID="0ee256a1-73bc-48e1-8912-e50ddd4320b1" containerID="75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a" exitCode=0 Aug 04 04:09:44.385590 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.385508 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" Aug 04 04:09:44.385590 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.385555 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" event={"ID":"0ee256a1-73bc-48e1-8912-e50ddd4320b1","Type":"ContainerDied","Data":"75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a"} Aug 04 04:09:44.385674 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.385594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb" event={"ID":"0ee256a1-73bc-48e1-8912-e50ddd4320b1","Type":"ContainerDied","Data":"02da4591597b3d5b3dd7c5092b9f5d2d9d1372a9955e131781cdd08040659644"} Aug 04 04:09:44.385674 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.385615 2568 scope.go:117] "RemoveContainer" containerID="75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a" Aug 04 04:09:44.396325 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.396307 2568 scope.go:117] "RemoveContainer" containerID="75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a" Aug 04 04:09:44.396593 ip-10-0-130-28 kubenswrapper[2568]: E0804 04:09:44.396575 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a\": container with ID starting with 75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a not found: ID does not exist" containerID="75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a" Aug 04 04:09:44.396644 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.396601 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a"} err="failed to get container status \"75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a\": rpc error: code = NotFound desc = could not find container \"75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a\": container with ID starting with 75b44596568ba586f1587df72fe9efe5731b9794bd49e69ac84feb48bb34a15a not found: ID does not exist" Aug 04 04:09:44.407547 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.407525 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb"] Aug 04 04:09:44.411520 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:44.411499 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-07c848/sub-test-model-07c848-kserve-6fc556964c-tt2sb"] Aug 04 04:09:45.373390 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:09:45.373312 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ee256a1-73bc-48e1-8912-e50ddd4320b1" path="/var/lib/kubelet/pods/0ee256a1-73bc-48e1-8912-e50ddd4320b1/volumes" Aug 04 04:10:03.188864 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:10:03.188832 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:10:58.981192 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:10:58.981111 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:11:13.590705 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:11:13.590672 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:11:43.313256 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:11:43.313226 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:11:57.692228 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:11:57.692195 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:12:24.498405 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:12:24.498365 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:12:54.299936 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:12:54.298604 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:13:24.090980 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:13:24.090941 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:13:56.687448 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:13:56.687415 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:14:07.561104 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.561065 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg"] Aug 04 04:14:07.561590 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.561573 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ee256a1-73bc-48e1-8912-e50ddd4320b1" containerName="main" Aug 04 04:14:07.561636 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.561593 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ee256a1-73bc-48e1-8912-e50ddd4320b1" containerName="main" Aug 04 04:14:07.561672 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.561655 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ee256a1-73bc-48e1-8912-e50ddd4320b1" containerName="main" Aug 04 04:14:07.565009 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.564988 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.567993 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.567976 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-feaa4d-openshift-default-dockercfg-xwkbb\"" Aug 04 04:14:07.576136 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.576116 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg"] Aug 04 04:14:07.673375 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-credential-socket\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673564 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-data\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673564 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673456 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-token\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673564 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673541 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-socket\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673738 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673738 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grshf\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-kube-api-access-grshf\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673738 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673651 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-envoy\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673917 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-certs\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.673917 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.673804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-podinfo\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775157 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-credential-socket\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775346 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775163 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-data\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775346 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-token\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775346 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-socket\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775521 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775521 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grshf\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-kube-api-access-grshf\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775521 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-envoy\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775521 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-certs\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775521 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775499 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-credential-socket\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775521 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-podinfo\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775806 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775562 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-data\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775806 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775752 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-socket\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.775875 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.775845 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-certs\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.776110 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.776091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.777585 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.777567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-envoy\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.778004 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.777988 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-podinfo\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.787353 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.787333 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grshf\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-kube-api-access-grshf\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.788992 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.788965 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-token\") pod \"e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:07.876226 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:07.876148 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:08.018150 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:08.018117 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg"] Aug 04 04:14:08.020839 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:14:08.020814 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605dd386_f4b8_4a7d_859f_ca24d57fd1f8.slice/crio-64d5d5c3efa6f903c14d7d7f208c525729332f50cb668f3344416bfbafffae75 WatchSource:0}: Error finding container 64d5d5c3efa6f903c14d7d7f208c525729332f50cb668f3344416bfbafffae75: Status 404 returned error can't find the container with id 64d5d5c3efa6f903c14d7d7f208c525729332f50cb668f3344416bfbafffae75 Aug 04 04:14:08.022936 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:08.022920 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 04:14:08.750114 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:08.750076 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" event={"ID":"605dd386-f4b8-4a7d-859f-ca24d57fd1f8","Type":"ContainerStarted","Data":"64d5d5c3efa6f903c14d7d7f208c525729332f50cb668f3344416bfbafffae75"} Aug 04 04:14:10.819444 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:10.819410 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 04:14:10.819723 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:10.819480 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 04:14:10.819723 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:10.819508 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 04:14:11.768984 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:11.768948 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" event={"ID":"605dd386-f4b8-4a7d-859f-ca24d57fd1f8","Type":"ContainerStarted","Data":"404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c"} Aug 04 04:14:11.792967 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:11.792919 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" podStartSLOduration=1.996921812 podStartE2EDuration="4.792903686s" totalImagesPullingTime="2.795981874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:14:11.790984713 +0000 UTC m=+3874.912054083" watchObservedRunningTime="2026-08-04 04:14:11.792903686 +0000 UTC m=+3874.913973019" Aug 04 04:14:11.876600 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:11.876571 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:11.881695 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:11.881673 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:12.774140 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:12.774104 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:12.775067 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:12.775046 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:27.897658 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:27.897614 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:14:36.901688 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:36.900528 2568 generic.go:356] "Generic (PLEG): container finished" podID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerID="404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c" exitCode=0 Aug 04 04:14:36.901688 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:36.900628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" event={"ID":"605dd386-f4b8-4a7d-859f-ca24d57fd1f8","Type":"ContainerDied","Data":"404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c"} Aug 04 04:14:36.901688 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:36.901096 2568 scope.go:117] "RemoveContainer" containerID="404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c" Aug 04 04:14:36.903804 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:36.903766 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 04:14:36.903919 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:36.903839 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 04:14:36.903919 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:36.903877 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 04:14:37.911159 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:37.911112 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" event={"ID":"605dd386-f4b8-4a7d-859f-ca24d57fd1f8","Type":"ContainerStarted","Data":"94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc"} Aug 04 04:14:38.876638 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:38.876602 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:38.877604 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:38.877581 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 04:14:38.877674 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:38.877630 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 04:14:39.554575 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.554537 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q"] Aug 04 04:14:39.557001 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.556985 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" Aug 04 04:14:39.559993 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.559972 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 04:14:39.560127 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.560051 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 04:14:39.565961 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.565938 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q"] Aug 04 04:14:39.576806 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.576778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513aef1d-8fc6-435c-ba99-17322d4513e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q\" (UID: \"513aef1d-8fc6-435c-ba99-17322d4513e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" Aug 04 04:14:39.677640 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.677601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513aef1d-8fc6-435c-ba99-17322d4513e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q\" (UID: \"513aef1d-8fc6-435c-ba99-17322d4513e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" Aug 04 04:14:39.678287 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.678269 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513aef1d-8fc6-435c-ba99-17322d4513e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q\" (UID: \"513aef1d-8fc6-435c-ba99-17322d4513e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" Aug 04 04:14:39.869261 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.869176 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" Aug 04 04:14:39.877495 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.877457 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 04:14:39.877616 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:39.877538 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 04:14:40.045134 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.045091 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q"] Aug 04 04:14:40.049600 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:14:40.049570 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513aef1d_8fc6_435c_ba99_17322d4513e0.slice/crio-355ea178a50b34184b35ad88f63949ccab991085b2ddceabc7884b4f94c2f4b7 WatchSource:0}: Error finding container 355ea178a50b34184b35ad88f63949ccab991085b2ddceabc7884b4f94c2f4b7: Status 404 returned error can't find the container with id 355ea178a50b34184b35ad88f63949ccab991085b2ddceabc7884b4f94c2f4b7 Aug 04 04:14:40.058032 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.058006 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:14:40.059343 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.059322 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:14:40.876691 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.876650 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Aug 04 04:14:40.877159 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.876710 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Aug 04 04:14:40.936176 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.936143 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" event={"ID":"513aef1d-8fc6-435c-ba99-17322d4513e0","Type":"ContainerStarted","Data":"d19f8e741d00c77133d67af71a8393a07ed067d4a6042c1e1f061f81e0db38a7"} Aug 04 04:14:40.936358 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.936248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" event={"ID":"513aef1d-8fc6-435c-ba99-17322d4513e0","Type":"ContainerStarted","Data":"355ea178a50b34184b35ad88f63949ccab991085b2ddceabc7884b4f94c2f4b7"} Aug 04 04:14:40.963805 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:40.963755 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" podStartSLOduration=1.963740237 podStartE2EDuration="1.963740237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:14:40.961672168 +0000 UTC m=+3904.082741529" watchObservedRunningTime="2026-08-04 04:14:40.963740237 +0000 UTC m=+3904.084809579" Aug 04 04:14:41.881161 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:41.881127 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:41.881548 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:41.881406 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:41.882142 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:41.882124 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:41.945619 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:41.945582 2568 generic.go:356] "Generic (PLEG): container finished" podID="513aef1d-8fc6-435c-ba99-17322d4513e0" containerID="d19f8e741d00c77133d67af71a8393a07ed067d4a6042c1e1f061f81e0db38a7" exitCode=0 Aug 04 04:14:41.946042 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:41.945886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" event={"ID":"513aef1d-8fc6-435c-ba99-17322d4513e0","Type":"ContainerDied","Data":"d19f8e741d00c77133d67af71a8393a07ed067d4a6042c1e1f061f81e0db38a7"} Aug 04 04:14:43.081986 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:43.081960 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" Aug 04 04:14:43.111372 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:43.111341 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513aef1d-8fc6-435c-ba99-17322d4513e0-maas-api-service-ca\") pod \"513aef1d-8fc6-435c-ba99-17322d4513e0\" (UID: \"513aef1d-8fc6-435c-ba99-17322d4513e0\") " Aug 04 04:14:43.111685 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:43.111665 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513aef1d-8fc6-435c-ba99-17322d4513e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "513aef1d-8fc6-435c-ba99-17322d4513e0" (UID: "513aef1d-8fc6-435c-ba99-17322d4513e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 04:14:43.212196 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:43.212112 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/513aef1d-8fc6-435c-ba99-17322d4513e0-maas-api-service-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:43.957195 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:43.957169 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" Aug 04 04:14:43.957366 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:43.957166 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q" event={"ID":"513aef1d-8fc6-435c-ba99-17322d4513e0","Type":"ContainerDied","Data":"355ea178a50b34184b35ad88f63949ccab991085b2ddceabc7884b4f94c2f4b7"} Aug 04 04:14:43.957366 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:43.957295 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="355ea178a50b34184b35ad88f63949ccab991085b2ddceabc7884b4f94c2f4b7" Aug 04 04:14:45.742367 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:45.742337 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q"] Aug 04 04:14:45.747701 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:45.747672 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-feaa4d-mls7q"] Aug 04 04:14:47.372522 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:47.372489 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513aef1d-8fc6-435c-ba99-17322d4513e0" path="/var/lib/kubelet/pods/513aef1d-8fc6-435c-ba99-17322d4513e0/volumes" Aug 04 04:14:49.788496 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:49.788462 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:14:52.517412 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:52.517378 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg"] Aug 04 04:14:52.517848 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:52.517651 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" containerID="cri-o://94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc" gracePeriod=30 Aug 04 04:14:56.150620 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.150583 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx"] Aug 04 04:14:56.151092 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.150984 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513aef1d-8fc6-435c-ba99-17322d4513e0" containerName="revoke-keys" Aug 04 04:14:56.151092 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.150996 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="513aef1d-8fc6-435c-ba99-17322d4513e0" containerName="revoke-keys" Aug 04 04:14:56.151092 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.151053 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="513aef1d-8fc6-435c-ba99-17322d4513e0" containerName="revoke-keys" Aug 04 04:14:56.154386 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.154371 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" Aug 04 04:14:56.157018 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.156994 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 04:14:56.157150 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.157051 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 04:14:56.161828 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.161804 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx"] Aug 04 04:14:56.223327 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.223290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5628e8-6f81-4e76-b696-0b33b5ec5cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx\" (UID: \"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" Aug 04 04:14:56.324336 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.324304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5628e8-6f81-4e76-b696-0b33b5ec5cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx\" (UID: \"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" Aug 04 04:14:56.324967 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.324948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5628e8-6f81-4e76-b696-0b33b5ec5cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx\" (UID: \"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" Aug 04 04:14:56.465775 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.465741 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" Aug 04 04:14:56.599455 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.599430 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx"] Aug 04 04:14:56.601478 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:14:56.601447 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5628e8_6f81_4e76_b696_0b33b5ec5cc8.slice/crio-13a5bc71fe46212d8e05734f3f4eabfba974f08a4e7b6e610366ff8540a562d5 WatchSource:0}: Error finding container 13a5bc71fe46212d8e05734f3f4eabfba974f08a4e7b6e610366ff8540a562d5: Status 404 returned error can't find the container with id 13a5bc71fe46212d8e05734f3f4eabfba974f08a4e7b6e610366ff8540a562d5 Aug 04 04:14:56.884532 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.884497 2568 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 04:14:56.884682 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:56.884553 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 04:14:57.028716 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.028683 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" event={"ID":"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8","Type":"ContainerStarted","Data":"8124cea8d47bf74d3540c4692f2044b6862d1be0c235a1f36217efc85e41d610"} Aug 04 04:14:57.028902 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.028735 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" event={"ID":"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8","Type":"ContainerStarted","Data":"13a5bc71fe46212d8e05734f3f4eabfba974f08a4e7b6e610366ff8540a562d5"} Aug 04 04:14:57.047451 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.047402 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" podStartSLOduration=1.047388729 podStartE2EDuration="1.047388729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:14:57.044851206 +0000 UTC m=+3920.165920573" watchObservedRunningTime="2026-08-04 04:14:57.047388729 +0000 UTC m=+3920.168458071" Aug 04 04:14:57.769518 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.769490 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:57.839352 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839319 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grshf\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-kube-api-access-grshf\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839515 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839363 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istiod-ca-cert\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839515 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839423 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-data\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839515 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839444 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-envoy\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839515 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839493 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-credential-socket\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839700 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839517 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-socket\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839700 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839551 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-certs\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839700 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839592 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-token\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839700 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839617 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-podinfo\") pod \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\" (UID: \"605dd386-f4b8-4a7d-859f-ca24d57fd1f8\") " Aug 04 04:14:57.839934 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839793 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:14:57.839934 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839885 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:14:57.840058 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839962 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-credential-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.840058 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839962 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:14:57.840058 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.839984 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 04:14:57.840671 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.840642 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:14:57.841824 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.841794 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 04:14:57.842015 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.841796 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 04:14:57.842015 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.841854 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 04:14:57.842015 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.841987 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-kube-api-access-grshf" (OuterVolumeSpecName: "kube-api-access-grshf") pod "605dd386-f4b8-4a7d-859f-ca24d57fd1f8" (UID: "605dd386-f4b8-4a7d-859f-ca24d57fd1f8"). InnerVolumeSpecName "kube-api-access-grshf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 04:14:57.941286 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941249 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istiod-ca-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.941286 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941280 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.941286 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941290 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-envoy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.941286 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941297 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.941608 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941305 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-workload-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.941608 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941315 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-token\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.941608 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941323 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-istio-podinfo\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:57.941608 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:57.941331 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grshf\" (UniqueName: \"kubernetes.io/projected/605dd386-f4b8-4a7d-859f-ca24d57fd1f8-kube-api-access-grshf\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:58.037092 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.037055 2568 generic.go:356] "Generic (PLEG): container finished" podID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerID="94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc" exitCode=0 Aug 04 04:14:58.037262 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.037109 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" Aug 04 04:14:58.037262 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.037140 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" event={"ID":"605dd386-f4b8-4a7d-859f-ca24d57fd1f8","Type":"ContainerDied","Data":"94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc"} Aug 04 04:14:58.037262 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.037198 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg" event={"ID":"605dd386-f4b8-4a7d-859f-ca24d57fd1f8","Type":"ContainerDied","Data":"64d5d5c3efa6f903c14d7d7f208c525729332f50cb668f3344416bfbafffae75"} Aug 04 04:14:58.037262 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.037228 2568 scope.go:117] "RemoveContainer" containerID="94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc" Aug 04 04:14:58.038764 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.038742 2568 generic.go:356] "Generic (PLEG): container finished" podID="4a5628e8-6f81-4e76-b696-0b33b5ec5cc8" containerID="8124cea8d47bf74d3540c4692f2044b6862d1be0c235a1f36217efc85e41d610" exitCode=0 Aug 04 04:14:58.038875 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.038823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" event={"ID":"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8","Type":"ContainerDied","Data":"8124cea8d47bf74d3540c4692f2044b6862d1be0c235a1f36217efc85e41d610"} Aug 04 04:14:58.047282 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.047262 2568 scope.go:117] "RemoveContainer" containerID="404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c" Aug 04 04:14:58.056331 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.056306 2568 scope.go:117] "RemoveContainer" containerID="94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc" Aug 04 04:14:58.056657 ip-10-0-130-28 kubenswrapper[2568]: E0804 04:14:58.056635 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc\": container with ID starting with 94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc not found: ID does not exist" containerID="94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc" Aug 04 04:14:58.056729 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.056666 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc"} err="failed to get container status \"94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc\": rpc error: code = NotFound desc = could not find container \"94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc\": container with ID starting with 94634187ee86fd529f77ba003636ed5bfbe96e5ef022774b22ab6a3487d116dc not found: ID does not exist" Aug 04 04:14:58.056729 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.056684 2568 scope.go:117] "RemoveContainer" containerID="404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c" Aug 04 04:14:58.056991 ip-10-0-130-28 kubenswrapper[2568]: E0804 04:14:58.056968 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c\": container with ID starting with 404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c not found: ID does not exist" containerID="404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c" Aug 04 04:14:58.057048 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.057003 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c"} err="failed to get container status \"404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c\": rpc error: code = NotFound desc = could not find container \"404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c\": container with ID starting with 404f9540a55ab9a29aa993b2cdda230a1b1234edfad2b9cb3b6af83110934d2c not found: ID does not exist" Aug 04 04:14:58.069877 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.069848 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg"] Aug 04 04:14:58.074688 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:58.074664 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-feaa4d-openshift-default-685ff788f8-bfbvg"] Aug 04 04:14:59.178745 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:59.178721 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" Aug 04 04:14:59.252232 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:59.252202 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5628e8-6f81-4e76-b696-0b33b5ec5cc8-maas-api-service-ca\") pod \"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8\" (UID: \"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8\") " Aug 04 04:14:59.252544 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:59.252522 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5628e8-6f81-4e76-b696-0b33b5ec5cc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a5628e8-6f81-4e76-b696-0b33b5ec5cc8" (UID: "4a5628e8-6f81-4e76-b696-0b33b5ec5cc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 04:14:59.352953 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:59.352843 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5628e8-6f81-4e76-b696-0b33b5ec5cc8-maas-api-service-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:14:59.373339 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:14:59.373311 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" path="/var/lib/kubelet/pods/605dd386-f4b8-4a7d-859f-ca24d57fd1f8/volumes" Aug 04 04:15:00.052876 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.052834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" event={"ID":"4a5628e8-6f81-4e76-b696-0b33b5ec5cc8","Type":"ContainerDied","Data":"13a5bc71fe46212d8e05734f3f4eabfba974f08a4e7b6e610366ff8540a562d5"} Aug 04 04:15:00.052876 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.052859 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx" Aug 04 04:15:00.053098 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.052914 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13a5bc71fe46212d8e05734f3f4eabfba974f08a4e7b6e610366ff8540a562d5" Aug 04 04:15:00.151932 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.151887 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89"] Aug 04 04:15:00.152278 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.152267 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" Aug 04 04:15:00.152322 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.152281 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" Aug 04 04:15:00.152322 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.152290 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5628e8-6f81-4e76-b696-0b33b5ec5cc8" containerName="revoke-keys" Aug 04 04:15:00.152322 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.152296 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5628e8-6f81-4e76-b696-0b33b5ec5cc8" containerName="revoke-keys" Aug 04 04:15:00.152443 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.152354 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" Aug 04 04:15:00.152443 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.152361 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5628e8-6f81-4e76-b696-0b33b5ec5cc8" containerName="revoke-keys" Aug 04 04:15:00.152443 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.152368 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" Aug 04 04:15:00.155731 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.155711 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq"] Aug 04 04:15:00.155888 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.155845 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" Aug 04 04:15:00.156092 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.156074 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" Aug 04 04:15:00.156151 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.156093 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="605dd386-f4b8-4a7d-859f-ca24d57fd1f8" containerName="istio-proxy" Aug 04 04:15:00.159017 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.158997 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2"] Aug 04 04:15:00.159139 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.159118 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" Aug 04 04:15:00.159227 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.159121 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 04:15:00.162249 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.162231 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg"] Aug 04 04:15:00.162362 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.162327 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" Aug 04 04:15:00.165785 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.165767 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd"] Aug 04 04:15:00.165907 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.165883 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" Aug 04 04:15:00.167516 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.167489 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" Aug 04 04:15:00.169620 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.169601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" Aug 04 04:15:00.169848 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.169829 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89"] Aug 04 04:15:00.174744 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.173532 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2"] Aug 04 04:15:00.177620 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.177331 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq"] Aug 04 04:15:00.179041 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.179018 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" Aug 04 04:15:00.180681 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.180664 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg"] Aug 04 04:15:00.188128 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.185304 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd"] Aug 04 04:15:00.216290 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.216261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" Aug 04 04:15:00.224726 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.224701 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" Aug 04 04:15:00.263077 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.262712 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" Aug 04 04:15:00.457531 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.457507 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd"] Aug 04 04:15:00.458844 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:15:00.458823 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7d1810_65cc_48a9_949e_a7ea98ce8437.slice/crio-53da99b416ff56a0ab9fd9eb90c67c7bea406c6862c87f253f5fb7586a178b25 WatchSource:0}: Error finding container 53da99b416ff56a0ab9fd9eb90c67c7bea406c6862c87f253f5fb7586a178b25: Status 404 returned error can't find the container with id 53da99b416ff56a0ab9fd9eb90c67c7bea406c6862c87f253f5fb7586a178b25 Aug 04 04:15:00.581771 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.581737 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq"] Aug 04 04:15:00.582907 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:15:00.582865 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5434c6_cd6f_459c_8300_869035eb82f4.slice/crio-aafa8b02b6e9d9ddcc96bfc83db819d99933eec084f0e949c9a07fde33f06d01 WatchSource:0}: Error finding container aafa8b02b6e9d9ddcc96bfc83db819d99933eec084f0e949c9a07fde33f06d01: Status 404 returned error can't find the container with id aafa8b02b6e9d9ddcc96bfc83db819d99933eec084f0e949c9a07fde33f06d01 Aug 04 04:15:00.583571 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:15:00.583480 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ed2b8d_77e3_4fbb_a279_70a0a4b9a16e.slice/crio-f5e17d241c7632f27a307ee0bd2a92c7146627004c16310d514a715fe81c9ff0 WatchSource:0}: Error finding container f5e17d241c7632f27a307ee0bd2a92c7146627004c16310d514a715fe81c9ff0: Status 404 returned error can't find the container with id f5e17d241c7632f27a307ee0bd2a92c7146627004c16310d514a715fe81c9ff0 Aug 04 04:15:00.584093 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.584073 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89"] Aug 04 04:15:00.623416 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.623353 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2"] Aug 04 04:15:00.623915 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:15:00.623875 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cebf261_b246_4b59_a81e_a9551d81f410.slice/crio-7e0f7479e7383b91613dc7acb18c5531cd90cb8a61943d338a36f840d074192f WatchSource:0}: Error finding container 7e0f7479e7383b91613dc7acb18c5531cd90cb8a61943d338a36f840d074192f: Status 404 returned error can't find the container with id 7e0f7479e7383b91613dc7acb18c5531cd90cb8a61943d338a36f840d074192f Aug 04 04:15:00.626445 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:00.626420 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg"] Aug 04 04:15:00.629889 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:15:00.629117 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cdbb38_90a6_4631_b975_cc5ea081c36c.slice/crio-d26b37f93466a906921deead22aa2fd4042875918f7ae2adeb4085ace6fc712d WatchSource:0}: Error finding container d26b37f93466a906921deead22aa2fd4042875918f7ae2adeb4085ace6fc712d: Status 404 returned error can't find the container with id d26b37f93466a906921deead22aa2fd4042875918f7ae2adeb4085ace6fc712d Aug 04 04:15:01.060244 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.060208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" event={"ID":"91cdbb38-90a6-4631-b975-cc5ea081c36c","Type":"ContainerStarted","Data":"87a2ec6be2cb126d60745b882e65d13134efb7e1e8563a353c3c655a3f19746f"} Aug 04 04:15:01.060402 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.060268 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" event={"ID":"91cdbb38-90a6-4631-b975-cc5ea081c36c","Type":"ContainerStarted","Data":"d26b37f93466a906921deead22aa2fd4042875918f7ae2adeb4085ace6fc712d"} Aug 04 04:15:01.061719 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.061694 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" event={"ID":"7cebf261-b246-4b59-a81e-a9551d81f410","Type":"ContainerStarted","Data":"dd86023abe8736109f528cf7041e6bb19bfbffb4a17aba46bffb53422a7fe517"} Aug 04 04:15:01.061839 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.061738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" event={"ID":"7cebf261-b246-4b59-a81e-a9551d81f410","Type":"ContainerStarted","Data":"7e0f7479e7383b91613dc7acb18c5531cd90cb8a61943d338a36f840d074192f"} Aug 04 04:15:01.063127 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.063100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" event={"ID":"71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e","Type":"ContainerStarted","Data":"8944a692c6666df558e9e21fa42dcbfd9469bf6a390aa8126cd903e8292a553a"} Aug 04 04:15:01.063256 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.063150 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" event={"ID":"71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e","Type":"ContainerStarted","Data":"f5e17d241c7632f27a307ee0bd2a92c7146627004c16310d514a715fe81c9ff0"} Aug 04 04:15:01.064619 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.064595 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" event={"ID":"5b5434c6-cd6f-459c-8300-869035eb82f4","Type":"ContainerStarted","Data":"83348b2fcb061bcd88439b6aa5b6d5416eeee236e46e4ea8aa6c0aae533a0353"} Aug 04 04:15:01.064738 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.064634 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" event={"ID":"5b5434c6-cd6f-459c-8300-869035eb82f4","Type":"ContainerStarted","Data":"aafa8b02b6e9d9ddcc96bfc83db819d99933eec084f0e949c9a07fde33f06d01"} Aug 04 04:15:01.066027 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.066007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" event={"ID":"bf7d1810-65cc-48a9-949e-a7ea98ce8437","Type":"ContainerStarted","Data":"ddb418c9281beaf858b91f1b9e905df9da5607411de423d96c0b616db48e6eb0"} Aug 04 04:15:01.066124 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.066040 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" event={"ID":"bf7d1810-65cc-48a9-949e-a7ea98ce8437","Type":"ContainerStarted","Data":"53da99b416ff56a0ab9fd9eb90c67c7bea406c6862c87f253f5fb7586a178b25"} Aug 04 04:15:01.078884 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.078844 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" podStartSLOduration=1.078829942 podStartE2EDuration="1.078829942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:15:01.077133126 +0000 UTC m=+3924.198202469" watchObservedRunningTime="2026-08-04 04:15:01.078829942 +0000 UTC m=+3924.199899263" Aug 04 04:15:01.092853 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.092813 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" podStartSLOduration=1.092801106 podStartE2EDuration="1.092801106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:15:01.092187916 +0000 UTC m=+3924.213274800" watchObservedRunningTime="2026-08-04 04:15:01.092801106 +0000 UTC m=+3924.213870447" Aug 04 04:15:01.109219 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.109172 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" podStartSLOduration=1.10916006 podStartE2EDuration="1.10916006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:15:01.107145476 +0000 UTC m=+3924.228214836" watchObservedRunningTime="2026-08-04 04:15:01.10916006 +0000 UTC m=+3924.230229396" Aug 04 04:15:01.125620 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.125564 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" podStartSLOduration=1.1255472659999999 podStartE2EDuration="1.125547266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:15:01.122709339 +0000 UTC m=+3924.243778694" watchObservedRunningTime="2026-08-04 04:15:01.125547266 +0000 UTC m=+3924.246616610" Aug 04 04:15:01.142950 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:01.142888 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" podStartSLOduration=1.142876297 podStartE2EDuration="1.142876297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:15:01.142037784 +0000 UTC m=+3924.263107126" watchObservedRunningTime="2026-08-04 04:15:01.142876297 +0000 UTC m=+3924.263945679" Aug 04 04:15:02.072978 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.072866 2568 generic.go:356] "Generic (PLEG): container finished" podID="91cdbb38-90a6-4631-b975-cc5ea081c36c" containerID="87a2ec6be2cb126d60745b882e65d13134efb7e1e8563a353c3c655a3f19746f" exitCode=0 Aug 04 04:15:02.072978 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.072927 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" event={"ID":"91cdbb38-90a6-4631-b975-cc5ea081c36c","Type":"ContainerDied","Data":"87a2ec6be2cb126d60745b882e65d13134efb7e1e8563a353c3c655a3f19746f"} Aug 04 04:15:02.074456 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.074433 2568 generic.go:356] "Generic (PLEG): container finished" podID="7cebf261-b246-4b59-a81e-a9551d81f410" containerID="dd86023abe8736109f528cf7041e6bb19bfbffb4a17aba46bffb53422a7fe517" exitCode=0 Aug 04 04:15:02.074567 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.074492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" event={"ID":"7cebf261-b246-4b59-a81e-a9551d81f410","Type":"ContainerDied","Data":"dd86023abe8736109f528cf7041e6bb19bfbffb4a17aba46bffb53422a7fe517"} Aug 04 04:15:02.075956 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.075935 2568 generic.go:356] "Generic (PLEG): container finished" podID="71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e" containerID="8944a692c6666df558e9e21fa42dcbfd9469bf6a390aa8126cd903e8292a553a" exitCode=0 Aug 04 04:15:02.076068 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.076047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" event={"ID":"71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e","Type":"ContainerDied","Data":"8944a692c6666df558e9e21fa42dcbfd9469bf6a390aa8126cd903e8292a553a"} Aug 04 04:15:02.077469 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.077448 2568 generic.go:356] "Generic (PLEG): container finished" podID="5b5434c6-cd6f-459c-8300-869035eb82f4" containerID="83348b2fcb061bcd88439b6aa5b6d5416eeee236e46e4ea8aa6c0aae533a0353" exitCode=0 Aug 04 04:15:02.077557 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.077523 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" event={"ID":"5b5434c6-cd6f-459c-8300-869035eb82f4","Type":"ContainerDied","Data":"83348b2fcb061bcd88439b6aa5b6d5416eeee236e46e4ea8aa6c0aae533a0353"} Aug 04 04:15:02.078907 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.078879 2568 generic.go:356] "Generic (PLEG): container finished" podID="bf7d1810-65cc-48a9-949e-a7ea98ce8437" containerID="ddb418c9281beaf858b91f1b9e905df9da5607411de423d96c0b616db48e6eb0" exitCode=0 Aug 04 04:15:02.079000 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:02.078951 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" event={"ID":"bf7d1810-65cc-48a9-949e-a7ea98ce8437","Type":"ContainerDied","Data":"ddb418c9281beaf858b91f1b9e905df9da5607411de423d96c0b616db48e6eb0"} Aug 04 04:15:03.224187 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:03.224158 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" Aug 04 04:15:03.268326 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:03.268297 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" Aug 04 04:15:03.305325 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:03.305300 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" Aug 04 04:15:03.313705 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:03.313682 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" Aug 04 04:15:03.317343 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:03.317325 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" Aug 04 04:15:04.091466 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.091432 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" Aug 04 04:15:04.091661 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.091466 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29763615-8wf89" event={"ID":"5b5434c6-cd6f-459c-8300-869035eb82f4","Type":"ContainerDied","Data":"aafa8b02b6e9d9ddcc96bfc83db819d99933eec084f0e949c9a07fde33f06d01"} Aug 04 04:15:04.091661 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.091525 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aafa8b02b6e9d9ddcc96bfc83db819d99933eec084f0e949c9a07fde33f06d01" Aug 04 04:15:04.092814 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.092793 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" Aug 04 04:15:04.092965 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.092833 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd" event={"ID":"bf7d1810-65cc-48a9-949e-a7ea98ce8437","Type":"ContainerDied","Data":"53da99b416ff56a0ab9fd9eb90c67c7bea406c6862c87f253f5fb7586a178b25"} Aug 04 04:15:04.092965 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.092876 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53da99b416ff56a0ab9fd9eb90c67c7bea406c6862c87f253f5fb7586a178b25" Aug 04 04:15:04.094298 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.094280 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" Aug 04 04:15:04.094422 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.094304 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg" event={"ID":"91cdbb38-90a6-4631-b975-cc5ea081c36c","Type":"ContainerDied","Data":"d26b37f93466a906921deead22aa2fd4042875918f7ae2adeb4085ace6fc712d"} Aug 04 04:15:04.094422 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.094343 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d26b37f93466a906921deead22aa2fd4042875918f7ae2adeb4085ace6fc712d" Aug 04 04:15:04.095873 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.095856 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" Aug 04 04:15:04.095978 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.095923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2" event={"ID":"7cebf261-b246-4b59-a81e-a9551d81f410","Type":"ContainerDied","Data":"7e0f7479e7383b91613dc7acb18c5531cd90cb8a61943d338a36f840d074192f"} Aug 04 04:15:04.095978 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.095969 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e0f7479e7383b91613dc7acb18c5531cd90cb8a61943d338a36f840d074192f" Aug 04 04:15:04.097373 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.097355 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" event={"ID":"71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e","Type":"ContainerDied","Data":"f5e17d241c7632f27a307ee0bd2a92c7146627004c16310d514a715fe81c9ff0"} Aug 04 04:15:04.097373 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.097370 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq" Aug 04 04:15:04.097478 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:04.097391 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5e17d241c7632f27a307ee0bd2a92c7146627004c16310d514a715fe81c9ff0" Aug 04 04:15:11.686711 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:11.686675 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg"] Aug 04 04:15:11.697329 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:11.697305 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-2d8d0c-29763615-kmtvg"] Aug 04 04:15:11.967957 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:11.967860 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx"] Aug 04 04:15:11.971481 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:11.971452 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d8d0c-fhnnx"] Aug 04 04:15:13.192047 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:13.192011 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:15:13.373830 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:13.373798 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5628e8-6f81-4e76-b696-0b33b5ec5cc8" path="/var/lib/kubelet/pods/4a5628e8-6f81-4e76-b696-0b33b5ec5cc8/volumes" Aug 04 04:15:13.374154 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:13.374139 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91cdbb38-90a6-4631-b975-cc5ea081c36c" path="/var/lib/kubelet/pods/91cdbb38-90a6-4631-b975-cc5ea081c36c/volumes" Aug 04 04:15:24.086029 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:15:24.085991 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:16:01.241632 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.241599 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln"] Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.241976 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91cdbb38-90a6-4631-b975-cc5ea081c36c" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.241990 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="91cdbb38-90a6-4631-b975-cc5ea081c36c" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242001 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cebf261-b246-4b59-a81e-a9551d81f410" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242006 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cebf261-b246-4b59-a81e-a9551d81f410" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242015 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242021 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242031 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7d1810-65cc-48a9-949e-a7ea98ce8437" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242036 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7d1810-65cc-48a9-949e-a7ea98ce8437" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242051 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5434c6-cd6f-459c-8300-869035eb82f4" containerName="cleanup" Aug 04 04:16:01.242053 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242057 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5434c6-cd6f-459c-8300-869035eb82f4" containerName="cleanup" Aug 04 04:16:01.242360 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242128 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cebf261-b246-4b59-a81e-a9551d81f410" containerName="cleanup" Aug 04 04:16:01.242360 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242138 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5434c6-cd6f-459c-8300-869035eb82f4" containerName="cleanup" Aug 04 04:16:01.242360 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242146 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7d1810-65cc-48a9-949e-a7ea98ce8437" containerName="cleanup" Aug 04 04:16:01.242360 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242156 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e" containerName="cleanup" Aug 04 04:16:01.242360 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.242163 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="91cdbb38-90a6-4631-b975-cc5ea081c36c" containerName="cleanup" Aug 04 04:16:01.245366 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.245351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" Aug 04 04:16:01.249352 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.249145 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 04:16:01.249352 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.249174 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 04:16:01.251012 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.250987 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln"] Aug 04 04:16:01.294150 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.294117 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109eaa95-0e42-41bb-9636-497f73285127-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln\" (UID: \"109eaa95-0e42-41bb-9636-497f73285127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" Aug 04 04:16:01.395361 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.395331 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109eaa95-0e42-41bb-9636-497f73285127-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln\" (UID: \"109eaa95-0e42-41bb-9636-497f73285127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" Aug 04 04:16:01.395918 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.395875 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109eaa95-0e42-41bb-9636-497f73285127-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln\" (UID: \"109eaa95-0e42-41bb-9636-497f73285127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" Aug 04 04:16:01.558349 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.558255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" Aug 04 04:16:01.693568 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:01.693538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln"] Aug 04 04:16:01.695226 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:16:01.695199 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109eaa95_0e42_41bb_9636_497f73285127.slice/crio-60a13f979d17b6e6dd72a09f927e623ca0571a641fac92c0a038912d99e676b9 WatchSource:0}: Error finding container 60a13f979d17b6e6dd72a09f927e623ca0571a641fac92c0a038912d99e676b9: Status 404 returned error can't find the container with id 60a13f979d17b6e6dd72a09f927e623ca0571a641fac92c0a038912d99e676b9 Aug 04 04:16:02.418347 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:02.418316 2568 generic.go:356] "Generic (PLEG): container finished" podID="109eaa95-0e42-41bb-9636-497f73285127" containerID="31ecbe10cd84c969e3ff11ea71c52c1ef98c7a2c0b7335f1f6841a1b6bb332f3" exitCode=0 Aug 04 04:16:02.418730 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:02.418386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" event={"ID":"109eaa95-0e42-41bb-9636-497f73285127","Type":"ContainerDied","Data":"31ecbe10cd84c969e3ff11ea71c52c1ef98c7a2c0b7335f1f6841a1b6bb332f3"} Aug 04 04:16:02.418730 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:02.418424 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" event={"ID":"109eaa95-0e42-41bb-9636-497f73285127","Type":"ContainerStarted","Data":"60a13f979d17b6e6dd72a09f927e623ca0571a641fac92c0a038912d99e676b9"} Aug 04 04:16:03.559283 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:03.559257 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" Aug 04 04:16:03.614089 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:03.614055 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109eaa95-0e42-41bb-9636-497f73285127-maas-api-service-ca\") pod \"109eaa95-0e42-41bb-9636-497f73285127\" (UID: \"109eaa95-0e42-41bb-9636-497f73285127\") " Aug 04 04:16:03.614411 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:03.614386 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109eaa95-0e42-41bb-9636-497f73285127-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "109eaa95-0e42-41bb-9636-497f73285127" (UID: "109eaa95-0e42-41bb-9636-497f73285127"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 04:16:03.715559 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:03.715471 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/109eaa95-0e42-41bb-9636-497f73285127-maas-api-service-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:16:04.432486 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:04.432453 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" event={"ID":"109eaa95-0e42-41bb-9636-497f73285127","Type":"ContainerDied","Data":"60a13f979d17b6e6dd72a09f927e623ca0571a641fac92c0a038912d99e676b9"} Aug 04 04:16:04.432486 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:04.432475 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln" Aug 04 04:16:04.432700 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:04.432509 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a13f979d17b6e6dd72a09f927e623ca0571a641fac92c0a038912d99e676b9" Aug 04 04:16:11.724467 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:11.724431 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq"] Aug 04 04:16:11.726616 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:11.726593 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b0089f-29763615-tqwqq"] Aug 04 04:16:11.896777 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:11.896746 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln"] Aug 04 04:16:11.899748 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:11.899726 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b0089f-g58ln"] Aug 04 04:16:13.373181 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:13.373152 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="109eaa95-0e42-41bb-9636-497f73285127" path="/var/lib/kubelet/pods/109eaa95-0e42-41bb-9636-497f73285127/volumes" Aug 04 04:16:13.373555 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:13.373432 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e" path="/var/lib/kubelet/pods/71ed2b8d-77e3-4fbb-a279-70a0a4b9a16e/volumes" Aug 04 04:16:15.296214 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:15.296180 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:16:18.798831 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.798790 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk"] Aug 04 04:16:18.799320 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.799304 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="109eaa95-0e42-41bb-9636-497f73285127" containerName="revoke-keys" Aug 04 04:16:18.799390 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.799324 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="109eaa95-0e42-41bb-9636-497f73285127" containerName="revoke-keys" Aug 04 04:16:18.799465 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.799451 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="109eaa95-0e42-41bb-9636-497f73285127" containerName="revoke-keys" Aug 04 04:16:18.803932 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.803911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" Aug 04 04:16:18.809228 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.809193 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 04:16:18.809418 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.809297 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 04:16:18.811354 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.811332 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk"] Aug 04 04:16:18.953727 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:18.953688 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk\" (UID: \"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" Aug 04 04:16:19.055014 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:19.054912 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk\" (UID: \"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" Aug 04 04:16:19.055468 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:19.055448 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk\" (UID: \"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" Aug 04 04:16:19.116627 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:19.116598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" Aug 04 04:16:19.248142 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:19.248108 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk"] Aug 04 04:16:19.248289 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:16:19.248270 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf8c9dc_e5a7_4615_b05c_4f9045ad5f01.slice/crio-92bf464262302efc262b5692bbb721968f02bb8bae7077d75567e124e4f2bb4c WatchSource:0}: Error finding container 92bf464262302efc262b5692bbb721968f02bb8bae7077d75567e124e4f2bb4c: Status 404 returned error can't find the container with id 92bf464262302efc262b5692bbb721968f02bb8bae7077d75567e124e4f2bb4c Aug 04 04:16:19.515432 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:19.515406 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" event={"ID":"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01","Type":"ContainerStarted","Data":"92bf464262302efc262b5692bbb721968f02bb8bae7077d75567e124e4f2bb4c"} Aug 04 04:16:20.522081 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:20.522046 2568 generic.go:356] "Generic (PLEG): container finished" podID="dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01" containerID="c338e54a240c63256dcf993125e3c52db5cc67590697352229ed3e4a7dd09a46" exitCode=0 Aug 04 04:16:20.522478 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:20.522130 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" event={"ID":"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01","Type":"ContainerDied","Data":"c338e54a240c63256dcf993125e3c52db5cc67590697352229ed3e4a7dd09a46"} Aug 04 04:16:21.668432 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:21.668408 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" Aug 04 04:16:21.780589 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:21.780556 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01-maas-api-service-ca\") pod \"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01\" (UID: \"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01\") " Aug 04 04:16:21.781004 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:21.780981 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01" (UID: "dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 04:16:21.881549 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:21.881464 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01-maas-api-service-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:16:22.539270 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:22.539241 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" Aug 04 04:16:22.539466 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:22.539271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk" event={"ID":"dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01","Type":"ContainerDied","Data":"92bf464262302efc262b5692bbb721968f02bb8bae7077d75567e124e4f2bb4c"} Aug 04 04:16:22.539466 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:22.539326 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92bf464262302efc262b5692bbb721968f02bb8bae7077d75567e124e4f2bb4c" Aug 04 04:16:29.545450 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:29.545407 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd"] Aug 04 04:16:29.549545 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:29.549515 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-07c848-29763615-g7wrd"] Aug 04 04:16:29.708507 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:29.708473 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk"] Aug 04 04:16:29.710748 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:29.710719 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-07c848-5jsjk"] Aug 04 04:16:31.374230 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:31.374190 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7d1810-65cc-48a9-949e-a7ea98ce8437" path="/var/lib/kubelet/pods/bf7d1810-65cc-48a9-949e-a7ea98ce8437/volumes" Aug 04 04:16:31.375218 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:31.375195 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01" path="/var/lib/kubelet/pods/dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01/volumes" Aug 04 04:16:33.084966 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:33.084929 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:16:36.600618 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.600580 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz"] Aug 04 04:16:36.601038 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.601022 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01" containerName="revoke-keys" Aug 04 04:16:36.601038 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.601039 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01" containerName="revoke-keys" Aug 04 04:16:36.601138 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.601102 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf8c9dc-e5a7-4615-b05c-4f9045ad5f01" containerName="revoke-keys" Aug 04 04:16:36.605432 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.605415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" Aug 04 04:16:36.608351 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.608316 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7lrf4\"" Aug 04 04:16:36.608488 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.608328 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 04:16:36.612996 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.612969 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz"] Aug 04 04:16:36.704956 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.704882 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz\" (UID: \"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" Aug 04 04:16:36.806264 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.806230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz\" (UID: \"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" Aug 04 04:16:36.806828 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.806810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz\" (UID: \"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" Aug 04 04:16:36.918327 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:36.918296 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" Aug 04 04:16:37.048692 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:37.048665 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz"] Aug 04 04:16:37.050522 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:16:37.050476 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4ab3dd_0e8e_49bb_b145_f1b2f3ca78d1.slice/crio-fc319b3520e6758eee1c8d305f907256b1cb842043da8a346a437c965bc036f6 WatchSource:0}: Error finding container fc319b3520e6758eee1c8d305f907256b1cb842043da8a346a437c965bc036f6: Status 404 returned error can't find the container with id fc319b3520e6758eee1c8d305f907256b1cb842043da8a346a437c965bc036f6 Aug 04 04:16:37.626681 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:37.626653 2568 generic.go:356] "Generic (PLEG): container finished" podID="bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1" containerID="01f9d68c7c225f2d7a83e5c92f90d6158a1943f1117a7fc4a0011386fd487421" exitCode=0 Aug 04 04:16:37.627017 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:37.626731 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" event={"ID":"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1","Type":"ContainerDied","Data":"01f9d68c7c225f2d7a83e5c92f90d6158a1943f1117a7fc4a0011386fd487421"} Aug 04 04:16:37.627017 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:37.626781 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" event={"ID":"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1","Type":"ContainerStarted","Data":"fc319b3520e6758eee1c8d305f907256b1cb842043da8a346a437c965bc036f6"} Aug 04 04:16:38.763544 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:38.763517 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" Aug 04 04:16:38.824972 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:38.824937 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1-maas-api-service-ca\") pod \"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1\" (UID: \"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1\") " Aug 04 04:16:38.825283 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:38.825260 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1" (UID: "bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 04:16:38.926005 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:38.925973 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1-maas-api-service-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\"" Aug 04 04:16:39.638920 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:39.638812 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" Aug 04 04:16:39.638920 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:39.638856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz" event={"ID":"bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1","Type":"ContainerDied","Data":"fc319b3520e6758eee1c8d305f907256b1cb842043da8a346a437c965bc036f6"} Aug 04 04:16:39.638920 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:39.638917 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc319b3520e6758eee1c8d305f907256b1cb842043da8a346a437c965bc036f6" Aug 04 04:16:47.234857 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:47.234819 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2"] Aug 04 04:16:47.238247 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:47.238224 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-881c6f-29763615-vjbd2"] Aug 04 04:16:47.373083 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:47.373048 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cebf261-b246-4b59-a81e-a9551d81f410" path="/var/lib/kubelet/pods/7cebf261-b246-4b59-a81e-a9551d81f410/volumes" Aug 04 04:16:47.422504 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:47.422470 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz"] Aug 04 04:16:47.425450 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:47.425426 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-881c6f-5vmvz"] Aug 04 04:16:49.373404 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:49.373373 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1" path="/var/lib/kubelet/pods/bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1/volumes" Aug 04 04:16:50.012829 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:16:50.012794 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gtnfh"] Aug 04 04:17:00.566764 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:00.566658 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2rtv6_be52a1d3-c577-422e-bb79-7852f540c3f0/manager/0.log" Aug 04 04:17:00.829230 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:00.829134 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-sng5x_37f7b958-9493-41bd-b931-9451f73b676f/copy-manifests/0.log" Aug 04 04:17:00.836187 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:00.836165 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-sng5x_37f7b958-9493-41bd-b931-9451f73b676f/manager/0.log" Aug 04 04:17:01.641439 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:01.641356 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29763615-8wf89_5b5434c6-cd6f-459c-8300-869035eb82f4/cleanup/0.log" Aug 04 04:17:02.526664 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.526630 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv_e724473a-29da-4b06-aeb5-a8929046f137/util/0.log" Aug 04 04:17:02.532199 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.532166 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv_e724473a-29da-4b06-aeb5-a8929046f137/pull/0.log" Aug 04 04:17:02.537479 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.537441 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv_e724473a-29da-4b06-aeb5-a8929046f137/extract/0.log" Aug 04 04:17:02.650250 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.650219 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n_36ad4170-0edc-490f-beac-3c807ea2da07/extract/0.log" Aug 04 04:17:02.657017 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.656989 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n_36ad4170-0edc-490f-beac-3c807ea2da07/util/0.log" Aug 04 04:17:02.662870 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.662847 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n_36ad4170-0edc-490f-beac-3c807ea2da07/pull/0.log" Aug 04 04:17:02.769124 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.769096 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd_38e874d0-ae52-409a-9ac3-21243b5aa4d6/util/0.log" Aug 04 04:17:02.775059 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.775035 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd_38e874d0-ae52-409a-9ac3-21243b5aa4d6/pull/0.log" Aug 04 04:17:02.781063 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.780991 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd_38e874d0-ae52-409a-9ac3-21243b5aa4d6/extract/0.log" Aug 04 04:17:02.895959 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.895928 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b_43031ab5-11a6-4b69-b581-2813b4db045f/util/0.log" Aug 04 04:17:02.902686 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.902663 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b_43031ab5-11a6-4b69-b581-2813b4db045f/pull/0.log" Aug 04 04:17:02.910258 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:02.909909 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b_43031ab5-11a6-4b69-b581-2813b4db045f/extract/0.log" Aug 04 04:17:03.740329 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:03.740303 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gtnfh_58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1/limitador/0.log" Aug 04 04:17:04.328737 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:04.328706 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6p4fp_99443321-7b05-4242-a9d5-1773be5407f3/discovery/0.log" Aug 04 04:17:04.992610 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:04.992576 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-885fb9fbb-7xtvr_42c1c927-755a-4827-99ca-58cd1cb90a46/router/0.log" Aug 04 04:17:05.677960 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:05.677931 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7tx7c_4d118ba8-88b5-4166-b99d-26860cc0c4c8/main/0.log" Aug 04 04:17:05.783019 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:05.782989 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-sct9k_2f065657-685a-4be9-81e1-7562b6095ef6/main/0.log" Aug 04 04:17:05.891046 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:05.891019 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-lfgv7_e8f1741b-5467-4df3-89f0-adbd1cbdfa17/main/0.log" Aug 04 04:17:14.445509 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:14.445462 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4k95v_379b75b7-a52e-4a10-be3d-89794b750ada/global-pull-secret-syncer/0.log" Aug 04 04:17:14.629508 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:14.629482 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tslnt_6c824696-e224-4701-a396-936afae4333d/konnectivity-agent/0.log" Aug 04 04:17:14.691104 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:14.691075 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-28.ec2.internal_fd8bacfae2b1c96a838e97cd988a77be/haproxy/0.log" Aug 04 04:17:19.442940 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.442888 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv_e724473a-29da-4b06-aeb5-a8929046f137/extract/0.log" Aug 04 04:17:19.469702 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.469678 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv_e724473a-29da-4b06-aeb5-a8929046f137/util/0.log" Aug 04 04:17:19.493328 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.493305 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595mtzv_e724473a-29da-4b06-aeb5-a8929046f137/pull/0.log" Aug 04 04:17:19.520135 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.520110 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n_36ad4170-0edc-490f-beac-3c807ea2da07/extract/0.log" Aug 04 04:17:19.544288 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.544262 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n_36ad4170-0edc-490f-beac-3c807ea2da07/util/0.log" Aug 04 04:17:19.570875 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.570853 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dqx2n_36ad4170-0edc-490f-beac-3c807ea2da07/pull/0.log" Aug 04 04:17:19.604174 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.604156 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd_38e874d0-ae52-409a-9ac3-21243b5aa4d6/extract/0.log" Aug 04 04:17:19.625208 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.625192 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd_38e874d0-ae52-409a-9ac3-21243b5aa4d6/util/0.log" Aug 04 04:17:19.648580 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.648563 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73c2rmd_38e874d0-ae52-409a-9ac3-21243b5aa4d6/pull/0.log" Aug 04 04:17:19.675984 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.675965 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b_43031ab5-11a6-4b69-b581-2813b4db045f/extract/0.log" Aug 04 04:17:19.706707 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.706648 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b_43031ab5-11a6-4b69-b581-2813b4db045f/util/0.log" Aug 04 04:17:19.728220 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:19.728200 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef18vp6b_43031ab5-11a6-4b69-b581-2813b4db045f/pull/0.log" Aug 04 04:17:20.552879 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:20.552843 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gtnfh_58bc4f81-2b90-4a4d-bcb6-9910aeff3dc1/limitador/0.log" Aug 04 04:17:22.637214 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:22.637173 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7cc6d447fc-nz69k_32ee0e56-7da3-4201-a0a1-d528f27dc836/metrics-server/0.log" Aug 04 04:17:22.663363 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:22.663335 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-fddjx_38897539-e1d5-4243-8589-1603f24557e9/monitoring-plugin/0.log" Aug 04 04:17:22.864039 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:22.863998 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpfhs_616ab198-16b9-4193-b6b2-522178036c5d/node-exporter/0.log" Aug 04 04:17:22.889223 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:22.889136 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpfhs_616ab198-16b9-4193-b6b2-522178036c5d/kube-rbac-proxy/0.log" Aug 04 04:17:22.917330 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:22.917305 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xpfhs_616ab198-16b9-4193-b6b2-522178036c5d/init-textfile/0.log" Aug 04 04:17:22.944758 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:22.944734 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-z786k_202ea3e0-881f-45b2-9a04-bdd8d46f58be/kube-rbac-proxy-main/0.log" Aug 04 04:17:22.975248 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:22.975208 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-z786k_202ea3e0-881f-45b2-9a04-bdd8d46f58be/kube-rbac-proxy-self/0.log" Aug 04 04:17:23.000658 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.000630 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-z786k_202ea3e0-881f-45b2-9a04-bdd8d46f58be/openshift-state-metrics/0.log" Aug 04 04:17:23.230369 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.230332 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-8m25v_996b9c63-9d9b-49e5-884c-2328d201d306/prometheus-operator/0.log" Aug 04 04:17:23.249913 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.249871 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-8m25v_996b9c63-9d9b-49e5-884c-2328d201d306/kube-rbac-proxy/0.log" Aug 04 04:17:23.385764 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.385734 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b44b789c-wnqsm_c838d367-f2be-4506-9d42-7193aa324e7f/thanos-query/0.log" Aug 04 04:17:23.406722 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.406701 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b44b789c-wnqsm_c838d367-f2be-4506-9d42-7193aa324e7f/kube-rbac-proxy-web/0.log" Aug 04 04:17:23.432048 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.432024 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b44b789c-wnqsm_c838d367-f2be-4506-9d42-7193aa324e7f/kube-rbac-proxy/0.log" Aug 04 04:17:23.459183 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.459155 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b44b789c-wnqsm_c838d367-f2be-4506-9d42-7193aa324e7f/prom-label-proxy/0.log" Aug 04 04:17:23.491752 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.491679 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b44b789c-wnqsm_c838d367-f2be-4506-9d42-7193aa324e7f/kube-rbac-proxy-rules/0.log" Aug 04 04:17:23.535600 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:23.535567 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55b44b789c-wnqsm_c838d367-f2be-4506-9d42-7193aa324e7f/kube-rbac-proxy-metrics/0.log" Aug 04 04:17:25.386218 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.386189 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/1.log" Aug 04 04:17:25.390283 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.390262 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-hkqrp_13504d71-ed32-4373-b189-966cf1a7f494/console-operator/2.log" Aug 04 04:17:25.460735 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.460703 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f"] Aug 04 04:17:25.461095 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.461081 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1" containerName="revoke-keys" Aug 04 04:17:25.461148 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.461097 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1" containerName="revoke-keys" Aug 04 04:17:25.461190 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.461161 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd4ab3dd-0e8e-49bb-b145-f1b2f3ca78d1" containerName="revoke-keys" Aug 04 04:17:25.464071 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.464055 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.466708 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.466693 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsl8p\"/\"openshift-service-ca.crt\"" Aug 04 04:17:25.467855 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.467833 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nsl8p\"/\"default-dockercfg-c855t\"" Aug 04 04:17:25.467978 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.467840 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsl8p\"/\"kube-root-ca.crt\"" Aug 04 04:17:25.474663 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.474641 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f"] Aug 04 04:17:25.538394 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.538359 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-lib-modules\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.538394 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.538398 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-sys\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.538614 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.538439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-proc\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.538614 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.538474 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-podres\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.538614 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.538522 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bj8z\" (UniqueName: \"kubernetes.io/projected/e3a6278a-959f-4339-b2ac-a68ca0e2062a-kube-api-access-9bj8z\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639528 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bj8z\" (UniqueName: \"kubernetes.io/projected/e3a6278a-959f-4339-b2ac-a68ca0e2062a-kube-api-access-9bj8z\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639665 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-lib-modules\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639665 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-sys\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639665 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639655 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-sys\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639837 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639703 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-lib-modules\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639837 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-proc\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639837 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639767 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-podres\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.639837 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639819 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-proc\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.640040 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.639928 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3a6278a-959f-4339-b2ac-a68ca0e2062a-podres\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.648204 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.648183 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bj8z\" (UniqueName: \"kubernetes.io/projected/e3a6278a-959f-4339-b2ac-a68ca0e2062a-kube-api-access-9bj8z\") pod \"perf-node-gather-daemonset-hn67f\" (UID: \"e3a6278a-959f-4339-b2ac-a68ca0e2062a\") " pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.774803 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.774770 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:25.907942 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:25.907917 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f"] Aug 04 04:17:25.909369 ip-10-0-130-28 kubenswrapper[2568]: W0804 04:17:25.909342 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode3a6278a_959f_4339_b2ac_a68ca0e2062a.slice/crio-e3b43a1829e9828a84d273995697e73991096325f615e7d6adf6071a245b70f5 WatchSource:0}: Error finding container e3b43a1829e9828a84d273995697e73991096325f615e7d6adf6071a245b70f5: Status 404 returned error can't find the container with id e3b43a1829e9828a84d273995697e73991096325f615e7d6adf6071a245b70f5 Aug 04 04:17:26.030497 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:26.030468 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-6t57s_3d38fb25-500a-4f20-884f-16cb59d68818/download-server/0.log" Aug 04 04:17:26.899875 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:26.899839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" event={"ID":"e3a6278a-959f-4339-b2ac-a68ca0e2062a","Type":"ContainerStarted","Data":"5fbbe7b5327b8f19da35d14654eafaf1e2485dfc3c3c73097634d9d6e9244e5f"} Aug 04 04:17:26.900277 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:26.899915 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" event={"ID":"e3a6278a-959f-4339-b2ac-a68ca0e2062a","Type":"ContainerStarted","Data":"e3b43a1829e9828a84d273995697e73991096325f615e7d6adf6071a245b70f5"} Aug 04 04:17:26.900277 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:26.900039 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:26.916812 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:26.916771 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" podStartSLOduration=1.916757305 podStartE2EDuration="1.916757305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 04:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 04:17:26.916163264 +0000 UTC m=+4070.037232623" watchObservedRunningTime="2026-08-04 04:17:26.916757305 +0000 UTC m=+4070.037826647" Aug 04 04:17:27.804674 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:27.804626 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tsqgz_cbe2b01b-b6ca-46f2-a852-c087b167d470/dns/0.log" Aug 04 04:17:27.825022 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:27.824998 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tsqgz_cbe2b01b-b6ca-46f2-a852-c087b167d470/kube-rbac-proxy/0.log" Aug 04 04:17:27.899855 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:27.899820 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vcc28_b2a765b1-015e-490f-8423-5779334c52a7/dns-node-resolver/0.log" Aug 04 04:17:28.483231 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:28.483203 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9vmv7_e2fb4df7-54b7-403d-bcbe-116f661d41c6/node-ca/0.log" Aug 04 04:17:29.608460 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:29.608426 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6p4fp_99443321-7b05-4242-a9d5-1773be5407f3/discovery/0.log" Aug 04 04:17:29.801818 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:29.801791 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-885fb9fbb-7xtvr_42c1c927-755a-4827-99ca-58cd1cb90a46/router/0.log" Aug 04 04:17:30.602407 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:30.602377 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mp8nz_4e8b9811-69f2-4b6a-ab51-1e200e549ff8/serve-healthcheck-canary/0.log" Aug 04 04:17:31.127862 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:31.127833 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-fvfjv_b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4/insights-operator/0.log" Aug 04 04:17:31.129020 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:31.128997 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-fvfjv_b8aab2c4-5624-4d78-bfd0-1a31d19ab9b4/insights-operator/1.log" Aug 04 04:17:31.224644 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:31.224615 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lxb82_870d694c-7f00-4f97-b9a4-258f94e1c31e/kube-rbac-proxy/0.log" Aug 04 04:17:31.249307 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:31.249280 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lxb82_870d694c-7f00-4f97-b9a4-258f94e1c31e/exporter/0.log" Aug 04 04:17:31.273650 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:31.273622 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lxb82_870d694c-7f00-4f97-b9a4-258f94e1c31e/extractor/0.log" Aug 04 04:17:32.914737 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:32.914710 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nsl8p/perf-node-gather-daemonset-hn67f" Aug 04 04:17:34.037478 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:34.037446 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2rtv6_be52a1d3-c577-422e-bb79-7852f540c3f0/manager/0.log" Aug 04 04:17:34.187723 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:34.187696 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-sng5x_37f7b958-9493-41bd-b931-9451f73b676f/manager/0.log" Aug 04 04:17:34.209310 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:34.209286 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6fb59c9fb6-sng5x_37f7b958-9493-41bd-b931-9451f73b676f/copy-manifests/0.log" Aug 04 04:17:35.739551 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:35.739521 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6488bdc59d-894hl_d192bd50-e074-452e-ad02-c67392401775/manager/0.log" Aug 04 04:17:35.804448 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:35.804421 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6p4fp_99443321-7b05-4242-a9d5-1773be5407f3/discovery/0.log" Aug 04 04:17:35.950746 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:35.950721 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-885fb9fbb-7xtvr_42c1c927-755a-4827-99ca-58cd1cb90a46/router/0.log" Aug 04 04:17:37.484208 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:37.484124 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-6p4fp_99443321-7b05-4242-a9d5-1773be5407f3/discovery/0.log" Aug 04 04:17:37.640500 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:37.640472 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-885fb9fbb-7xtvr_42c1c927-755a-4827-99ca-58cd1cb90a46/router/0.log" Aug 04 04:17:41.219740 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:41.219707 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-srg2h_b9f2069c-9b92-451c-ac80-087ecf892a3d/kube-storage-version-migrator-operator/1.log" Aug 04 04:17:41.220675 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:41.220658 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-srg2h_b9f2069c-9b92-451c-ac80-087ecf892a3d/kube-storage-version-migrator-operator/0.log" Aug 04 04:17:42.221798 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.221759 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7dpjj_cee1d932-e8b1-4374-8d76-9c98e6c35d7a/kube-multus/0.log" Aug 04 04:17:42.262303 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.262277 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4v9nw_5ff0d620-1bad-46a0-afcd-7cc8bcddccbc/kube-multus-additional-cni-plugins/0.log" Aug 04 04:17:42.282882 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.282863 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4v9nw_5ff0d620-1bad-46a0-afcd-7cc8bcddccbc/egress-router-binary-copy/0.log" Aug 04 04:17:42.304021 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.304004 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4v9nw_5ff0d620-1bad-46a0-afcd-7cc8bcddccbc/cni-plugins/0.log" Aug 04 04:17:42.327805 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.327789 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4v9nw_5ff0d620-1bad-46a0-afcd-7cc8bcddccbc/bond-cni-plugin/0.log" Aug 04 04:17:42.350082 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.350062 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4v9nw_5ff0d620-1bad-46a0-afcd-7cc8bcddccbc/routeoverride-cni/0.log" Aug 04 04:17:42.369846 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.369825 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4v9nw_5ff0d620-1bad-46a0-afcd-7cc8bcddccbc/whereabouts-cni-bincopy/0.log" Aug 04 04:17:42.398815 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.398792 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4v9nw_5ff0d620-1bad-46a0-afcd-7cc8bcddccbc/whereabouts-cni/0.log" Aug 04 04:17:42.929097 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.929068 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lgtdn_ed8629cd-8988-40bb-a461-2155943874ca/network-metrics-daemon/0.log" Aug 04 04:17:42.952098 ip-10-0-130-28 kubenswrapper[2568]: I0804 04:17:42.952075 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lgtdn_ed8629cd-8988-40bb-a461-2155943874ca/kube-rbac-proxy/0.log"