Sep 10 02:33:09.575292 ip-10-0-143-228 systemd[1]: Starting Kubernetes Kubelet... Sep 10 02:33:10.018434 ip-10-0-143-228 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 02:33:10.018434 ip-10-0-143-228 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 02:33:10.018434 ip-10-0-143-228 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 02:33:10.018434 ip-10-0-143-228 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 02:33:10.018434 ip-10-0-143-228 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 02:33:10.019625 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.019535 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 02:33:10.023857 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023841 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023858 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023864 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023868 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023871 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023875 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023878 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023882 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023886 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023889 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023892 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023895 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023897 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023900 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023903 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023905 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023908 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:33:10.023903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023911 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023914 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023917 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023920 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023923 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023925 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023929 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023931 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023936 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023939 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023942 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023952 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023954 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023957 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023960 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023975 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023978 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023981 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023983 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023986 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:33:10.024364 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023989 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023991 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023994 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023997 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.023999 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024002 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024004 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024007 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024009 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024012 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024014 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024017 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024019 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024022 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024025 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024028 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024031 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024033 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024036 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024039 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:33:10.024850 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024041 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024044 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024046 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024049 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024051 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024054 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024056 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024059 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024061 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024064 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024067 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024069 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024073 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024076 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024079 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024082 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024085 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024088 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024091 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024093 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:33:10.025355 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024097 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024100 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024103 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024105 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024108 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024111 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024114 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024117 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024120 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024123 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024125 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024128 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024131 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024133 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024136 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024138 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024141 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024143 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024146 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024148 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:33:10.025844 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024151 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024154 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024846 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024852 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024855 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024858 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024861 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024864 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024867 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024869 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024872 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024875 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024878 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024880 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024883 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024886 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024889 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024892 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024895 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024898 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:33:10.026374 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024900 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024903 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024906 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024909 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024911 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024914 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024917 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024919 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024923 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024927 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024930 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024933 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024935 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024938 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024940 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024943 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024946 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024948 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024951 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:33:10.026903 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024954 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024956 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024959 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024976 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024979 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024982 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024984 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024989 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024992 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.024995 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025000 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025005 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025007 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025010 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025013 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025015 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025018 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025020 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025023 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025026 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:33:10.027412 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025028 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025031 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025033 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025036 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025039 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025041 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025044 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025047 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025049 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025052 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025055 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025057 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025060 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025062 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025064 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025068 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025071 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025073 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025076 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025078 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:33:10.027896 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025081 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025083 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025087 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025090 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025093 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025095 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025098 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025101 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025103 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025107 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025110 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025113 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025116 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025119 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025121 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025124 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025127 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025130 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025133 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:33:10.028399 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025135 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025138 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025140 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025524 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025528 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025531 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025534 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025536 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025539 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025541 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025544 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025547 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025549 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025552 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025554 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025557 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025561 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025563 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025568 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025571 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:33:10.028856 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025575 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025577 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025581 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025583 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025587 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025590 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025593 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025596 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025598 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025601 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025603 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025606 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025608 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025611 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025613 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025617 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025621 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025624 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025626 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025629 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:33:10.029462 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025631 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025634 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025636 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025639 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025641 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025644 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025647 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025649 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025652 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025655 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025658 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025660 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025663 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025665 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025667 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025670 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025673 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025675 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025677 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025680 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:33:10.029958 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025682 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025685 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025689 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025692 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025695 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025698 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025701 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025704 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025707 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025710 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025713 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025716 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025719 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025721 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025724 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025726 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025729 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025732 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025734 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:33:10.030469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025737 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025740 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025742 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025745 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025748 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025750 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025752 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025755 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025758 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025760 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025763 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025766 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025768 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025771 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025773 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025776 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025778 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025781 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025786 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025789 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:33:10.030936 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025793 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025795 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.025798 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025871 2570 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025904 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025917 2570 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025922 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025926 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025929 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025933 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025937 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025940 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025943 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025947 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025951 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025954 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025976 2570 flags.go:64] FLAG: --cgroup-root="" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025980 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025983 2570 flags.go:64] FLAG: --client-ca-file="" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025986 2570 flags.go:64] FLAG: --cloud-provider="external" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025989 2570 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025995 2570 flags.go:64] FLAG: --cluster-domain="" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.025998 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026001 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 02:33:10.031506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026004 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026007 2570 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026011 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026014 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026017 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026020 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026027 2570 flags.go:64] FLAG: --contention-profiling="false" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026030 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026033 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026036 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026038 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026042 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026045 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026048 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026051 2570 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026054 2570 flags.go:64] FLAG: --enable-server="true" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026057 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026062 2570 flags.go:64] FLAG: --event-burst="100" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026065 2570 flags.go:64] FLAG: --event-qps="50" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026068 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026071 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026074 2570 flags.go:64] FLAG: --eviction-hard="" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026078 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026081 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026083 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 02:33:10.032210 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026090 2570 flags.go:64] FLAG: --eviction-soft="" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026093 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026096 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026099 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026101 2570 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026104 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026107 2570 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026110 2570 flags.go:64] FLAG: --feature-gates="" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026113 2570 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026116 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026119 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026122 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026125 2570 flags.go:64] FLAG: --healthz-port="10248" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026129 2570 flags.go:64] FLAG: --help="false" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026132 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026136 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026139 2570 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026142 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026152 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026156 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026159 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026162 2570 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026165 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 02:33:10.032934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026168 2570 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026171 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026174 2570 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026177 2570 flags.go:64] FLAG: --kube-reserved="" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026180 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026183 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026186 2570 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026189 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026192 2570 flags.go:64] FLAG: --lock-file="" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026195 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026198 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026202 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026207 2570 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026210 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026213 2570 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026216 2570 flags.go:64] FLAG: --logging-format="text" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026219 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026222 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026225 2570 flags.go:64] FLAG: --manifest-url="" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026228 2570 flags.go:64] FLAG: --manifest-url-header="" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026232 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026235 2570 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026241 2570 flags.go:64] FLAG: --max-pods="110" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026244 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026247 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 02:33:10.033609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026250 2570 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026253 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026256 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026259 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026262 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026269 2570 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026272 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026275 2570 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026279 2570 flags.go:64] FLAG: --pod-cidr="" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026281 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026287 2570 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026289 2570 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026293 2570 flags.go:64] FLAG: --pods-per-core="0" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026295 2570 flags.go:64] FLAG: --port="10250" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026299 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026302 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d897ab80ed7fa837" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026305 2570 flags.go:64] FLAG: --qos-reserved="" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026308 2570 flags.go:64] FLAG: --read-only-port="10255" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026315 2570 flags.go:64] FLAG: --register-node="true" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026318 2570 flags.go:64] FLAG: --register-with-taints="" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026322 2570 flags.go:64] FLAG: --registry-burst="10" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026325 2570 flags.go:64] FLAG: --registry-qps="5" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026328 2570 flags.go:64] FLAG: --reserved-cpus="" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026330 2570 flags.go:64] FLAG: --reserved-memory="" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026334 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 02:33:10.034330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026338 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026341 2570 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026344 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026346 2570 flags.go:64] FLAG: --runonce="false" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026349 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026353 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026356 2570 flags.go:64] FLAG: --seccomp-default="false" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026359 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026362 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026365 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026368 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026371 2570 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026375 2570 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026378 2570 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026381 2570 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026384 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026387 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026390 2570 flags.go:64] FLAG: --system-cgroups="" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026393 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026398 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026401 2570 flags.go:64] FLAG: --tls-cert-file="" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026404 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026411 2570 flags.go:64] FLAG: --tls-min-version="" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026414 2570 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026416 2570 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 02:33:10.035028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026419 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026428 2570 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026432 2570 flags.go:64] FLAG: --v="2" Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026436 2570 flags.go:64] FLAG: --version="false" Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026440 2570 flags.go:64] FLAG: --vmodule="" Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026443 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.026447 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026564 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026568 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026571 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026574 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026579 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026584 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026588 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026591 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026594 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026597 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026599 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026602 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026605 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026608 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:33:10.035687 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026611 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026613 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026616 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026618 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026621 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026624 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026626 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026629 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026631 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026634 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026636 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026639 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026642 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026649 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026651 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026654 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026657 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026659 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026663 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026666 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:33:10.036302 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026669 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026671 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026675 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026679 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026682 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026685 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026687 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026690 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026693 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026695 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026698 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026700 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026703 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026705 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026708 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026710 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026713 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026715 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026718 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026721 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:33:10.036833 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026723 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026726 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026728 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026731 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026733 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026736 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026741 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026744 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026747 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026750 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026752 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026754 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026757 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026760 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026764 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026767 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026770 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026773 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026776 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026778 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:33:10.037336 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026781 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026783 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026786 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026789 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026791 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026794 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026796 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026799 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026801 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026804 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026806 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026809 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026811 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026815 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026819 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026821 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026824 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026827 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026830 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:33:10.037818 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026838 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026841 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026844 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026847 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026849 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.026852 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.027779 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.035787 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.035802 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.036706 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.036718 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036781 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036786 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036790 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:33:10.038324 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036793 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036796 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036799 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036802 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036805 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036808 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036811 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036814 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036817 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036820 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036822 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036825 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036828 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036830 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036833 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036836 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036838 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036841 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036844 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036846 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:33:10.038681 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036849 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036851 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036854 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036857 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036859 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036864 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036868 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036871 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036874 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036877 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036879 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036882 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036884 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036887 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036892 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036896 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036899 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036902 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036904 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036907 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:33:10.039185 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036910 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036912 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036915 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036918 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036920 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036923 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036926 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036928 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036931 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036933 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036936 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036938 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036941 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036943 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036946 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036948 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036952 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036956 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036959 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:33:10.039686 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036982 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036986 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036989 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036991 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036994 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.036997 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037000 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037002 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037005 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037007 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037010 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037012 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037016 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037019 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037022 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037025 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037028 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037031 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037033 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037036 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:33:10.040251 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037038 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037041 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037043 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037046 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037048 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037051 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037054 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037056 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037059 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037061 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037064 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037067 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037070 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037079 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037082 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037085 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:33:10.040760 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037087 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.037093 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037265 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037270 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037274 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037277 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037280 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037283 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037285 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037289 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037293 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037296 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037299 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037302 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 02:33:10.041162 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037304 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037307 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037310 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037312 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037315 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037317 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037320 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037322 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037325 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037327 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037329 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037332 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037335 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037337 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037341 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037343 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037352 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037355 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037358 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037360 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 02:33:10.041504 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037363 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037365 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037367 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037370 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037373 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037375 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037377 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037380 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037382 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037385 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037387 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037390 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037392 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037395 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037397 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037400 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037402 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037404 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037407 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037410 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 02:33:10.042021 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037412 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037415 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037418 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037420 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037423 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037426 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037428 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037431 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037434 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037442 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037445 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037448 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037450 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037453 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037455 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037458 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037460 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037463 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037465 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037468 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 02:33:10.042510 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037470 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037474 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037477 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037480 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037482 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037485 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037488 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037492 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037495 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037498 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037501 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037504 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037507 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037509 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037512 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037514 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037518 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037521 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037524 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 02:33:10.043036 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037526 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037529 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037531 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037540 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037543 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037545 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037548 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:10.037550 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.037555 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.037686 2570 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.040953 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.042057 2570 server.go:1019] "Starting client certificate rotation" Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.042181 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 02:33:10.043495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.042222 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 02:33:10.047010 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.046989 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 02:33:10.052637 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.052615 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 02:33:10.069135 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.069114 2570 log.go:25] "Validated CRI v1 runtime API" Sep 10 02:33:10.073502 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.073482 2570 log.go:25] "Validated CRI v1 image API" Sep 10 02:33:10.074272 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.074260 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 02:33:10.077550 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.077525 2570 fs.go:135] Filesystem UUIDs: map[3b6e02b6-7473-4b8e-89f9-51f1d69cde9e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fdb1f55d-1756-48ee-8018-b7c23dfb47e8:/dev/nvme0n1p3] Sep 10 02:33:10.077605 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.077550 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 02:33:10.079304 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.079284 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 02:33:10.083307 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.083188 2570 manager.go:216] Machine: {Timestamp:2026-09-10 02:33:10.081558884 +0000 UTC m=+0.379517043 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3134560 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2071b42002cd3d1fc767ece525278e SystemUUID:ec2071b4-2002-cd3d-1fc7-67ece525278e BootID:efc1fd06-ac20-4282-932a-29d74163ab0f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:c3:34:7a:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:c3:34:7a:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:21:80:27:43:f4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 02:33:10.083307 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.083301 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 02:33:10.083423 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.083411 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 02:33:10.086196 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.086173 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 02:33:10.086336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.086198 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-228.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 02:33:10.087102 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.087092 2570 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 02:33:10.087135 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.087104 2570 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 02:33:10.087135 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.087112 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 02:33:10.088127 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.088117 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 02:33:10.090449 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.090433 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 02:33:10.095153 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095141 2570 state_mem.go:36] "Initialized new in-memory state store" Sep 10 02:33:10.095279 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095269 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 02:33:10.095315 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095307 2570 kubelet.go:486] "Attempting to sync node with API server" Sep 10 02:33:10.095346 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095328 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 02:33:10.095381 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095350 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 02:33:10.095381 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095365 2570 kubelet.go:398] "Adding apiserver pod source" Sep 10 02:33:10.095449 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095417 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 02:33:10.095893 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095873 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 02:33:10.095893 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.095895 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 02:33:10.100919 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.100892 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 02:33:10.103905 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.103891 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 02:33:10.103978 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.103921 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 02:33:10.104118 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104101 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 02:33:10.104204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104135 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 02:33:10.104204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104145 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 02:33:10.104204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104151 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 02:33:10.104204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104159 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 02:33:10.104204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104164 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 02:33:10.104773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104762 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 02:33:10.104773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104772 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 02:33:10.104833 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104779 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 02:33:10.104833 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104785 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 02:33:10.104833 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104794 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 02:33:10.104833 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.104802 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 02:33:10.105670 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.105655 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 02:33:10.105670 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.105669 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 02:33:10.105880 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.105856 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 02:33:10.105953 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.105931 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 02:33:10.109406 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.109391 2570 server.go:1268] "Started kubelet" Sep 10 02:33:10.109497 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.109450 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 02:33:10.109976 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.109909 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 02:33:10.110043 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.110001 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 02:33:10.110342 ip-10-0-143-228 systemd[1]: Started Kubernetes Kubelet. Sep 10 02:33:10.111612 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.111527 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 02:33:10.111612 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.111591 2570 server.go:310] "Adding debug handlers to kubelet server" Sep 10 02:33:10.114129 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.114006 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 02:33:10.117468 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.117453 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-228.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 02:33:10.118665 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.117530 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-228.ec2.internal.18d3d4ecb4f2447f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-228.ec2.internal,UID:ip-10-0-143-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-228.ec2.internal,},FirstTimestamp:2026-09-10 02:33:10.109365375 +0000 UTC m=+0.407323532,LastTimestamp:2026-09-10 02:33:10.109365375 +0000 UTC m=+0.407323532,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-228.ec2.internal,}" Sep 10 02:33:10.118828 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.118813 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 02:33:10.119489 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.119474 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 02:33:10.120569 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.120550 2570 factory.go:55] Registering systemd factory Sep 10 02:33:10.120569 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.120570 2570 factory.go:223] Registration of the systemd container factory successfully Sep 10 02:33:10.121171 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.121134 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:10.121228 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.121213 2570 factory.go:153] Registering CRI-O factory Sep 10 02:33:10.121269 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.121228 2570 factory.go:223] Registration of the crio container factory successfully Sep 10 02:33:10.121298 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.121289 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 02:33:10.121327 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.121307 2570 factory.go:103] Registering Raw factory Sep 10 02:33:10.121354 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.121338 2570 manager.go:1196] Started watching for new ooms in manager Sep 10 02:33:10.121401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.121390 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 02:33:10.121861 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.121846 2570 manager.go:349] Starting recovery of all containers Sep 10 02:33:10.123139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.123119 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 02:33:10.123230 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.123179 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 02:33:10.123282 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.123227 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6qdmx" Sep 10 02:33:10.123502 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.123471 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-228.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 02:33:10.123665 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.123619 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 02:33:10.123906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.123891 2570 reconstruct.go:97] "Volume reconstruction finished" Sep 10 02:33:10.123948 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.123905 2570 reconciler.go:29] "Reconciler: start to sync state" Sep 10 02:33:10.130418 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.130297 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6qdmx" Sep 10 02:33:10.135701 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.135685 2570 manager.go:354] Recovery completed Sep 10 02:33:10.139753 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.139740 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:33:10.141509 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.141493 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:33:10.141592 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.141526 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:33:10.141592 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.141543 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:33:10.142204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.142181 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 02:33:10.142204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.142194 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 02:33:10.142308 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.142210 2570 state_mem.go:36] "Initialized new in-memory state store" Sep 10 02:33:10.142308 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.142210 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:33:10.143001 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.142946 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:33:10.143064 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.143008 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:33:10.143064 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.143022 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:33:10.143196 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.143116 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-228.ec2.internal.18d3d4ecb6dcc115 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-228.ec2.internal,UID:ip-10-0-143-228.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-228.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-228.ec2.internal,},FirstTimestamp:2026-09-10 02:33:10.141509909 +0000 UTC m=+0.439468071,LastTimestamp:2026-09-10 02:33:10.141509909 +0000 UTC m=+0.439468071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-228.ec2.internal,}" Sep 10 02:33:10.144239 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.144226 2570 policy_none.go:49] "None policy: Start" Sep 10 02:33:10.144310 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.144242 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 02:33:10.144310 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.144252 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 02:33:10.146195 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.146182 2570 policy_none.go:47] "Start" Sep 10 02:33:10.182573 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.182556 2570 manager.go:337] "Starting Device Plugin manager" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.182602 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.182617 2570 server.go:87] "Starting device plugin registration server" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.182935 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.182948 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.183023 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.183092 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.183100 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.183648 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.183729 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.188027 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.189222 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.189276 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.189301 2570 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.189343 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 02:33:10.198139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.191577 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:33:10.283173 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.283074 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:33:10.284092 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.284073 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:33:10.284172 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.284108 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:33:10.284172 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.284124 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:33:10.284239 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.284179 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.289695 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.289655 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal"] Sep 10 02:33:10.289766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.289753 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:33:10.290677 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.290662 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:33:10.290745 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.290689 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:33:10.290745 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.290699 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:33:10.293066 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.293042 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:33:10.293191 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.293179 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.293255 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.293207 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:33:10.294271 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.294259 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.294318 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.294277 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-228.ec2.internal\": node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:10.294587 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.294567 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:33:10.294587 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.294578 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:33:10.294587 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.294593 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:33:10.294757 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.294598 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:33:10.294757 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.294604 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:33:10.294757 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.294616 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:33:10.296893 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.296879 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.296979 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.296904 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 02:33:10.298616 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.298599 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientMemory" Sep 10 02:33:10.298704 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.298625 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 02:33:10.298704 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.298634 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeHasSufficientPID" Sep 10 02:33:10.319787 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.319765 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:10.324834 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.324816 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-228.ec2.internal\" not found" node="ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.325618 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.325598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4729ed97ef7a8c38fc61d9b5517096f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"e4729ed97ef7a8c38fc61d9b5517096f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.325697 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.325624 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4729ed97ef7a8c38fc61d9b5517096f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"e4729ed97ef7a8c38fc61d9b5517096f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.325697 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.325652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/728b04a33548859b274bbec2cc5b3fee-config\") pod \"kube-apiserver-proxy-ip-10-0-143-228.ec2.internal\" (UID: \"728b04a33548859b274bbec2cc5b3fee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.329497 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.329483 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-228.ec2.internal\" not found" node="ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.420128 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.420092 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:10.426485 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.426456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4729ed97ef7a8c38fc61d9b5517096f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"e4729ed97ef7a8c38fc61d9b5517096f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.426610 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.426488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4729ed97ef7a8c38fc61d9b5517096f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"e4729ed97ef7a8c38fc61d9b5517096f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.426610 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.426550 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4729ed97ef7a8c38fc61d9b5517096f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"e4729ed97ef7a8c38fc61d9b5517096f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.426610 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.426547 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e4729ed97ef7a8c38fc61d9b5517096f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal\" (UID: \"e4729ed97ef7a8c38fc61d9b5517096f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.426720 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.426607 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/728b04a33548859b274bbec2cc5b3fee-config\") pod \"kube-apiserver-proxy-ip-10-0-143-228.ec2.internal\" (UID: \"728b04a33548859b274bbec2cc5b3fee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.426720 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.426608 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/728b04a33548859b274bbec2cc5b3fee-config\") pod \"kube-apiserver-proxy-ip-10-0-143-228.ec2.internal\" (UID: \"728b04a33548859b274bbec2cc5b3fee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.520652 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.520607 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:10.621421 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:10.621350 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:10.628671 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.628653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:10.634078 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:10.634000 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" Sep 10 02:33:11.754320 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:11.754297 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:11.854887 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:11.854842 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:11.955486 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:11.955405 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:12.055981 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:12.055954 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-228.ec2.internal\" not found" Sep 10 02:33:12.074303 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.074267 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 02:33:12.074463 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.074434 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 02:33:12.149267 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.149239 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:33:12.151155 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.151137 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 02:33:12.152603 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.152588 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" Sep 10 02:33:12.163642 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.163615 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 02:33:12.165558 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.165514 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 02:28:11 +0000 UTC" deadline="2028-06-15 13:46:13.619743402 +0000 UTC" Sep 10 02:33:12.165558 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.165554 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15467h13m1.454192432s" Sep 10 02:33:12.167009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.166994 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 02:33:12.168967 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.168951 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" Sep 10 02:33:12.178231 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.178216 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 02:33:12.185628 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.185609 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hfkx4" Sep 10 02:33:12.193560 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.193543 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hfkx4" Sep 10 02:33:12.218979 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:12.218953 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728b04a33548859b274bbec2cc5b3fee.slice/crio-e3cfbce1ce76d071e009b6a632480c9780ebfdca467130f57a23cd5c99a9f5e4 WatchSource:0}: Error finding container e3cfbce1ce76d071e009b6a632480c9780ebfdca467130f57a23cd5c99a9f5e4: Status 404 returned error can't find the container with id e3cfbce1ce76d071e009b6a632480c9780ebfdca467130f57a23cd5c99a9f5e4 Sep 10 02:33:12.219750 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:12.219727 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4729ed97ef7a8c38fc61d9b5517096f.slice/crio-963faede4e30cf02dd9aba21b6918d3979d383b81f138e99c38dc9ab3222b899 WatchSource:0}: Error finding container 963faede4e30cf02dd9aba21b6918d3979d383b81f138e99c38dc9ab3222b899: Status 404 returned error can't find the container with id 963faede4e30cf02dd9aba21b6918d3979d383b81f138e99c38dc9ab3222b899 Sep 10 02:33:12.224633 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.224610 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 02:33:12.224800 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.224758 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" event={"ID":"e4729ed97ef7a8c38fc61d9b5517096f","Type":"ContainerStarted","Data":"963faede4e30cf02dd9aba21b6918d3979d383b81f138e99c38dc9ab3222b899"} Sep 10 02:33:12.225953 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.225933 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" event={"ID":"728b04a33548859b274bbec2cc5b3fee","Type":"ContainerStarted","Data":"e3cfbce1ce76d071e009b6a632480c9780ebfdca467130f57a23cd5c99a9f5e4"} Sep 10 02:33:12.243939 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.243918 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:33:12.630876 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:12.630780 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:33:13.127747 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.127661 2570 apiserver.go:52] "Watching apiserver" Sep 10 02:33:13.136081 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.136055 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 02:33:13.136562 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.136537 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-8xc2b","openshift-multus/multus-additional-cni-plugins-87cr5","openshift-multus/network-metrics-daemon-6hhxs","openshift-cluster-node-tuning-operator/tuned-m5zpn","openshift-dns/node-resolver-gvvrb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal","openshift-network-diagnostics/network-check-target-9rzqt","openshift-network-operator/iptables-alerter-4z6sw","openshift-ovn-kubernetes/ovnkube-node-rs6vw","kube-system/global-pull-secret-syncer-sg8dz","kube-system/konnectivity-agent-bmppx","kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj","openshift-image-registry/node-ca-w7fk8"] Sep 10 02:33:13.139189 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.139165 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:13.139303 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.139253 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:13.141257 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.141238 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.143438 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.143412 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:13.143542 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.143492 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:13.143837 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.143779 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 02:33:13.143837 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.143782 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 02:33:13.144006 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.143853 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6zcsq\"" Sep 10 02:33:13.144006 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.143919 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 02:33:13.145645 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.145626 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.148106 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.147813 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:33:13.148106 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.147883 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skq9b\"" Sep 10 02:33:13.148106 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.147971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.148313 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.148118 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 02:33:13.150021 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.149989 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 02:33:13.150840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.150162 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7lnzb\"" Sep 10 02:33:13.150840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.150280 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 02:33:13.150840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.150415 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.152465 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.152436 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 02:33:13.152621 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.152599 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 02:33:13.152708 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.152692 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 02:33:13.153190 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.153037 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hb847\"" Sep 10 02:33:13.153374 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.153354 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.155748 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.155730 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 02:33:13.156258 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.156242 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 02:33:13.156440 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.156423 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 02:33:13.156626 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.156602 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 02:33:13.156806 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.156787 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c4sh9\"" Sep 10 02:33:13.157004 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.156986 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 02:33:13.157188 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.157172 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 02:33:13.157387 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.157369 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.158763 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.158452 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.159675 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.159605 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 02:33:13.161141 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.161121 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 02:33:13.161615 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.161408 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 02:33:13.161615 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.161509 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tdgc2\"" Sep 10 02:33:13.161737 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.161663 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 02:33:13.161943 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.161921 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 02:33:13.162166 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.162150 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 02:33:13.162683 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.162665 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-z7php\"" Sep 10 02:33:13.165206 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.165186 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.165290 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.165266 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:13.167374 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167352 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-device-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.167462 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167406 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-kubernetes\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.167462 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167444 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysctl-conf\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.167577 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167480 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-run\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.167577 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167506 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-cni-bin\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.167577 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-systemd-units\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.167577 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167570 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-modprobe-d\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.167851 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167593 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-kubelet\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.167851 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167723 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-system-cni-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.167851 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-daemon-config\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.167851 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-socket-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-kubelet\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-cni-bin\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167914 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-env-overrides\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167929 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167943 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-sys-fs\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167957 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-slash\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167974 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-log-socket\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.167996 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-conf-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-cnibin\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168043 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-etc-kubernetes\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-systemd\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-var-lib-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-sys\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55541690-c514-404a-8037-7bb94a66eaf5-hosts-file\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c45f\" (UniqueName: \"kubernetes.io/projected/8458a415-6e5d-497d-8b39-bfa2aa005fea-kube-api-access-8c45f\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168197 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168220 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55541690-c514-404a-8037-7bb94a66eaf5-tmp-dir\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-etc-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168268 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-registration-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-host\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-lib-modules\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkf4m\" (UniqueName: \"kubernetes.io/projected/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-kube-api-access-xkf4m\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168343 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9t5\" (UniqueName: \"kubernetes.io/projected/55541690-c514-404a-8037-7bb94a66eaf5-kube-api-access-5r9t5\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168359 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-cni-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.168400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-run-netns\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-ovnkube-script-lib\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-netns\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-var-lib-kubelet\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168581 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca08812a-7a23-4716-8d22-e116d09a39a4-ovn-node-metrics-cert\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-k8s-cni-cncf-io\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168699 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168728 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-tmp\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168752 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-ovn\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168804 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-cni-binary-copy\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfn2f\" (UniqueName: \"kubernetes.io/projected/253d3186-27d1-4b7e-b48a-303871be8896-kube-api-access-vfn2f\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168928 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-cni-netd\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.168977 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjcc\" (UniqueName: \"kubernetes.io/projected/ca08812a-7a23-4716-8d22-e116d09a39a4-kube-api-access-2sjcc\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169001 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-socket-dir-parent\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.169185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169024 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-cni-multus\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169048 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysconfig\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysctl-d\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169106 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-tuned\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169128 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-node-log\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-ovnkube-config\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169173 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-serviceca\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169197 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvbm\" (UniqueName: \"kubernetes.io/projected/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-kube-api-access-jvvbm\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169219 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-systemd\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-host\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169275 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-hostroot\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-multus-certs\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.169895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.169334 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfrx\" (UniqueName: \"kubernetes.io/projected/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-kube-api-access-8nfrx\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.171223 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.171162 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.171634 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.171560 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 02:33:13.171634 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.171594 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 02:33:13.172010 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.171951 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4f9cf\"" Sep 10 02:33:13.173498 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.173476 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 02:33:13.173498 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.173482 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 02:33:13.173863 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.173671 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mq2tf\"" Sep 10 02:33:13.173863 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.173721 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:33:13.196499 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.196444 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:28:12 +0000 UTC" deadline="2028-02-11 23:34:05.355005957 +0000 UTC" Sep 10 02:33:13.196644 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.196630 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12477h0m52.158381301s" Sep 10 02:33:13.254679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.254649 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 02:33:13.269737 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-serviceca\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.269944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvbm\" (UniqueName: \"kubernetes.io/projected/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-kube-api-access-jvvbm\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.269944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-systemd\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.269944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269806 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-host\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.269944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-hostroot\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.269944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269884 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-multus-certs\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.269944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269911 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfrx\" (UniqueName: \"kubernetes.io/projected/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-kube-api-access-8nfrx\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.269944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-systemd\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-device-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.269985 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-device-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270012 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270046 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-host\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-kubernetes\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-hostroot\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysctl-conf\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-run\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270143 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-multus-certs\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-cni-bin\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270202 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3902b22c-0dda-4604-a7d8-439602edd83e-dbus\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e758b-e536-410e-902e-864dd206dabd-konnectivity-ca\") pod \"konnectivity-agent-bmppx\" (UID: \"8e1e758b-e536-410e-902e-864dd206dabd\") " pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysctl-conf\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270314 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-serviceca\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-systemd-units\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-modprobe-d\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270406 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-kubernetes\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270425 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-cni-bin\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-run\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270447 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-systemd-units\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270467 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-modprobe-d\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270481 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-kubelet\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270508 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3902b22c-0dda-4604-a7d8-439602edd83e-kubelet-config\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270530 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-kubelet\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270532 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-cnibin\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270579 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfh7\" (UniqueName: \"kubernetes.io/projected/85a8b7bf-b25c-49fc-8889-80653c50b02b-kube-api-access-mpfh7\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270619 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-system-cni-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270644 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-daemon-config\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-socket-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270693 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.270974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270722 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-kubelet\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-cni-bin\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-env-overrides\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270874 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-sys-fs\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270899 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69zk\" (UniqueName: \"kubernetes.io/projected/d280979a-b322-40d6-b312-4ae92bb42883-kube-api-access-b69zk\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-slash\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-log-socket\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.270982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-conf-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-system-cni-dir\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-cni-binary-copy\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271100 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-cnibin\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-etc-kubernetes\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271154 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-kubelet\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271182 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271167 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-system-cni-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.271766 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-systemd\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271245 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-conf-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271253 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-systemd\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-var-lib-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-sys\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271324 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-etc-kubernetes\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55541690-c514-404a-8037-7bb94a66eaf5-hosts-file\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55541690-c514-404a-8037-7bb94a66eaf5-hosts-file\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271374 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c45f\" (UniqueName: \"kubernetes.io/projected/8458a415-6e5d-497d-8b39-bfa2aa005fea-kube-api-access-8c45f\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271134 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-cni-bin\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271410 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-socket-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271466 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271471 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-cnibin\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55541690-c514-404a-8037-7bb94a66eaf5-tmp-dir\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.271547 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271577 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-etc-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.272552 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.271611 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:13.77158986 +0000 UTC m=+3.037686261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271628 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-var-lib-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271664 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-sys-fs\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271677 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-slash\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271547 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-etc-openvswitch\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271699 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-sys\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-registration-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271740 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-host\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271766 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e1e758b-e536-410e-902e-864dd206dabd-agent-certs\") pod \"konnectivity-agent-bmppx\" (UID: \"8e1e758b-e536-410e-902e-864dd206dabd\") " pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271778 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-env-overrides\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-lib-modules\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271790 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-daemon-config\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkf4m\" (UniqueName: \"kubernetes.io/projected/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-kube-api-access-xkf4m\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271821 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55541690-c514-404a-8037-7bb94a66eaf5-tmp-dir\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271857 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-log-socket\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271870 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9t5\" (UniqueName: \"kubernetes.io/projected/55541690-c514-404a-8037-7bb94a66eaf5-kube-api-access-5r9t5\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271906 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-host\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271919 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-registration-dir\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.273401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271920 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-cni-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.271980 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-cni-dir\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272023 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-lib-modules\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d280979a-b322-40d6-b312-4ae92bb42883-iptables-alerter-script\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d280979a-b322-40d6-b312-4ae92bb42883-host-slash\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-run-netns\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-ovnkube-script-lib\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-netns\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272189 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-var-lib-kubelet\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272202 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-netns\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca08812a-7a23-4716-8d22-e116d09a39a4-ovn-node-metrics-cert\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272163 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-run-netns\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-k8s-cni-cncf-io\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272288 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-var-lib-kubelet\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-tmp\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.274130 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-ovn\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272338 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-cni-binary-copy\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfn2f\" (UniqueName: \"kubernetes.io/projected/253d3186-27d1-4b7e-b48a-303871be8896-kube-api-access-vfn2f\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-run-k8s-cni-cncf-io\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272318 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8458a415-6e5d-497d-8b39-bfa2aa005fea-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-cni-netd\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272456 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjcc\" (UniqueName: \"kubernetes.io/projected/ca08812a-7a23-4716-8d22-e116d09a39a4-kube-api-access-2sjcc\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-socket-dir-parent\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272525 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-cni-multus\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272560 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysconfig\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272571 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-run-ovn\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysctl-d\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-tuned\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272619 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-ovnkube-script-lib\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.274906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272619 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-node-log\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272649 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-node-log\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-ovnkube-config\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-multus-socket-dir-parent\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272739 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysconfig\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-run-ovn-kubernetes\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272811 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-sysctl-d\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272868 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca08812a-7a23-4716-8d22-e116d09a39a4-host-cni-netd\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272911 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-host-var-lib-cni-multus\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.272947 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.273295 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca08812a-7a23-4716-8d22-e116d09a39a4-ovnkube-config\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.275691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.273938 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-cni-binary-copy\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.276651 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.276539 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-etc-tuned\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.276752 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.276602 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-tmp\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.276802 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.276758 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca08812a-7a23-4716-8d22-e116d09a39a4-ovn-node-metrics-cert\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.292132 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.292101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9t5\" (UniqueName: \"kubernetes.io/projected/55541690-c514-404a-8037-7bb94a66eaf5-kube-api-access-5r9t5\") pod \"node-resolver-gvvrb\" (UID: \"55541690-c514-404a-8037-7bb94a66eaf5\") " pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.292266 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.292204 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfrx\" (UniqueName: \"kubernetes.io/projected/14a6ac2d-65fb-4070-b0f0-979edbcf7aa9-kube-api-access-8nfrx\") pod \"multus-8xc2b\" (UID: \"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9\") " pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.292462 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.292444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvbm\" (UniqueName: \"kubernetes.io/projected/1c19c39b-51ec-4ed1-8be8-f99fc93448a5-kube-api-access-jvvbm\") pod \"node-ca-w7fk8\" (UID: \"1c19c39b-51ec-4ed1-8be8-f99fc93448a5\") " pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.297906 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.297883 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:33:13.297906 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.297909 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:33:13.298095 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.297923 2570 projected.go:196] Error preparing data for projected volume kube-api-access-gvd6b for pod openshift-network-diagnostics/network-check-target-9rzqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:13.298095 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.297990 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b podName:6c26c64a-148b-430a-b8a3-26ab98b62904 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:13.797968123 +0000 UTC m=+3.064064505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gvd6b" (UniqueName: "kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b") pod "network-check-target-9rzqt" (UID: "6c26c64a-148b-430a-b8a3-26ab98b62904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:13.298368 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.298343 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkf4m\" (UniqueName: \"kubernetes.io/projected/9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4-kube-api-access-xkf4m\") pod \"tuned-m5zpn\" (UID: \"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4\") " pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.299493 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.299454 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c45f\" (UniqueName: \"kubernetes.io/projected/8458a415-6e5d-497d-8b39-bfa2aa005fea-kube-api-access-8c45f\") pod \"aws-ebs-csi-driver-node-qtnkj\" (UID: \"8458a415-6e5d-497d-8b39-bfa2aa005fea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.300033 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.299987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfn2f\" (UniqueName: \"kubernetes.io/projected/253d3186-27d1-4b7e-b48a-303871be8896-kube-api-access-vfn2f\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:13.300720 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.300699 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjcc\" (UniqueName: \"kubernetes.io/projected/ca08812a-7a23-4716-8d22-e116d09a39a4-kube-api-access-2sjcc\") pod \"ovnkube-node-rs6vw\" (UID: \"ca08812a-7a23-4716-8d22-e116d09a39a4\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.373842 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.373788 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e1e758b-e536-410e-902e-864dd206dabd-agent-certs\") pod \"konnectivity-agent-bmppx\" (UID: \"8e1e758b-e536-410e-902e-864dd206dabd\") " pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.374025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.373850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d280979a-b322-40d6-b312-4ae92bb42883-iptables-alerter-script\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.374025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.373878 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d280979a-b322-40d6-b312-4ae92bb42883-host-slash\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.374025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.373920 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.374025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.373956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.373983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3902b22c-0dda-4604-a7d8-439602edd83e-dbus\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.374025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e758b-e536-410e-902e-864dd206dabd-konnectivity-ca\") pod \"konnectivity-agent-bmppx\" (UID: \"8e1e758b-e536-410e-902e-864dd206dabd\") " pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.374025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d280979a-b322-40d6-b312-4ae92bb42883-host-slash\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374088 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3902b22c-0dda-4604-a7d8-439602edd83e-kubelet-config\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-cnibin\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfh7\" (UniqueName: \"kubernetes.io/projected/85a8b7bf-b25c-49fc-8889-80653c50b02b-kube-api-access-mpfh7\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374179 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69zk\" (UniqueName: \"kubernetes.io/projected/d280979a-b322-40d6-b312-4ae92bb42883-kube-api-access-b69zk\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374243 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-cnibin\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374271 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3902b22c-0dda-4604-a7d8-439602edd83e-kubelet-config\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374397 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3902b22c-0dda-4604-a7d8-439602edd83e-dbus\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374425 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-system-cni-dir\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374451 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-cni-binary-copy\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.374490 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.374546 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret podName:3902b22c-0dda-4604-a7d8-439602edd83e nodeName:}" failed. No retries permitted until 2026-09-10 02:33:13.874527935 +0000 UTC m=+3.140624310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret") pod "global-pull-secret-syncer-sg8dz" (UID: "3902b22c-0dda-4604-a7d8-439602edd83e") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374552 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d280979a-b322-40d6-b312-4ae92bb42883-iptables-alerter-script\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374611 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.374760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374664 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a8b7bf-b25c-49fc-8889-80653c50b02b-system-cni-dir\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.375131 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374860 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e758b-e536-410e-902e-864dd206dabd-konnectivity-ca\") pod \"konnectivity-agent-bmppx\" (UID: \"8e1e758b-e536-410e-902e-864dd206dabd\") " pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.375131 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.374926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-cni-binary-copy\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.375778 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.375748 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a8b7bf-b25c-49fc-8889-80653c50b02b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.376741 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.376695 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e1e758b-e536-410e-902e-864dd206dabd-agent-certs\") pod \"konnectivity-agent-bmppx\" (UID: \"8e1e758b-e536-410e-902e-864dd206dabd\") " pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.383769 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.383707 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfh7\" (UniqueName: \"kubernetes.io/projected/85a8b7bf-b25c-49fc-8889-80653c50b02b-kube-api-access-mpfh7\") pod \"multus-additional-cni-plugins-87cr5\" (UID: \"85a8b7bf-b25c-49fc-8889-80653c50b02b\") " pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.383769 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.383749 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69zk\" (UniqueName: \"kubernetes.io/projected/d280979a-b322-40d6-b312-4ae92bb42883-kube-api-access-b69zk\") pod \"iptables-alerter-4z6sw\" (UID: \"d280979a-b322-40d6-b312-4ae92bb42883\") " pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.430732 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.430702 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 02:33:13.451981 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.451933 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" Sep 10 02:33:13.463597 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.463561 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" Sep 10 02:33:13.473413 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.473391 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gvvrb" Sep 10 02:33:13.480678 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.480659 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w7fk8" Sep 10 02:33:13.489496 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.489472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:13.494205 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.494184 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xc2b" Sep 10 02:33:13.502713 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.502693 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-87cr5" Sep 10 02:33:13.509390 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.509373 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:13.516243 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.516221 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4z6sw" Sep 10 02:33:13.777491 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.777400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:13.777648 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.777568 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:13.777648 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.777639 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:14.777620848 +0000 UTC m=+4.043717220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:13.878463 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.878428 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:13.878463 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:13.878468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:13.878680 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.878593 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:13.878680 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.878611 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:33:13.878680 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.878640 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:33:13.878680 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.878650 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret podName:3902b22c-0dda-4604-a7d8-439602edd83e nodeName:}" failed. No retries permitted until 2026-09-10 02:33:14.878635178 +0000 UTC m=+4.144731547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret") pod "global-pull-secret-syncer-sg8dz" (UID: "3902b22c-0dda-4604-a7d8-439602edd83e") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:13.878680 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.878655 2570 projected.go:196] Error preparing data for projected volume kube-api-access-gvd6b for pod openshift-network-diagnostics/network-check-target-9rzqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:13.878899 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:13.878707 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b podName:6c26c64a-148b-430a-b8a3-26ab98b62904 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:14.878690413 +0000 UTC m=+4.144786796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvd6b" (UniqueName: "kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b") pod "network-check-target-9rzqt" (UID: "6c26c64a-148b-430a-b8a3-26ab98b62904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:14.177013 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.176986 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca08812a_7a23_4716_8d22_e116d09a39a4.slice/crio-d99d0a3731caf962af1b9c2dbe6f831430a31b4d9e823f3858aefd32a4f91571 WatchSource:0}: Error finding container d99d0a3731caf962af1b9c2dbe6f831430a31b4d9e823f3858aefd32a4f91571: Status 404 returned error can't find the container with id d99d0a3731caf962af1b9c2dbe6f831430a31b4d9e823f3858aefd32a4f91571 Sep 10 02:33:14.178715 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.178682 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55541690_c514_404a_8037_7bb94a66eaf5.slice/crio-fbd1917c04ae9cf247db99d73c83c1a25e178c5cb007f9def16ca637fd42405c WatchSource:0}: Error finding container fbd1917c04ae9cf247db99d73c83c1a25e178c5cb007f9def16ca637fd42405c: Status 404 returned error can't find the container with id fbd1917c04ae9cf247db99d73c83c1a25e178c5cb007f9def16ca637fd42405c Sep 10 02:33:14.181868 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.181810 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd280979a_b322_40d6_b312_4ae92bb42883.slice/crio-ccd54aa3c487b7a4119ae377f5d2b409888b114bd75b22dfda43ab9af73b491d WatchSource:0}: Error finding container ccd54aa3c487b7a4119ae377f5d2b409888b114bd75b22dfda43ab9af73b491d: Status 404 returned error can't find the container with id ccd54aa3c487b7a4119ae377f5d2b409888b114bd75b22dfda43ab9af73b491d Sep 10 02:33:14.182836 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.182797 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec9e53c_6f18_4d3e_af62_2bcce14dc7d4.slice/crio-ac34d877e631e95fdf8fcfe8dc9cf8d16bbf1ef1fe9b9826d71492aac6299054 WatchSource:0}: Error finding container ac34d877e631e95fdf8fcfe8dc9cf8d16bbf1ef1fe9b9826d71492aac6299054: Status 404 returned error can't find the container with id ac34d877e631e95fdf8fcfe8dc9cf8d16bbf1ef1fe9b9826d71492aac6299054 Sep 10 02:33:14.183481 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.183408 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1e758b_e536_410e_902e_864dd206dabd.slice/crio-bd66bf83c0cb6a9072e5a826468b60827cd30c4ca6cc239806e0b5762fdff37f WatchSource:0}: Error finding container bd66bf83c0cb6a9072e5a826468b60827cd30c4ca6cc239806e0b5762fdff37f: Status 404 returned error can't find the container with id bd66bf83c0cb6a9072e5a826468b60827cd30c4ca6cc239806e0b5762fdff37f Sep 10 02:33:14.184101 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.184079 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c19c39b_51ec_4ed1_8be8_f99fc93448a5.slice/crio-1b36c7748ef61d255e3bd48d2bbdb2a2778fbb108eba98acb2731165c4ddea1c WatchSource:0}: Error finding container 1b36c7748ef61d255e3bd48d2bbdb2a2778fbb108eba98acb2731165c4ddea1c: Status 404 returned error can't find the container with id 1b36c7748ef61d255e3bd48d2bbdb2a2778fbb108eba98acb2731165c4ddea1c Sep 10 02:33:14.185895 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.185221 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a6ac2d_65fb_4070_b0f0_979edbcf7aa9.slice/crio-6883f6613f6f61d62224baff22542df2967dc4df8f84e414f8f317cc273b3226 WatchSource:0}: Error finding container 6883f6613f6f61d62224baff22542df2967dc4df8f84e414f8f317cc273b3226: Status 404 returned error can't find the container with id 6883f6613f6f61d62224baff22542df2967dc4df8f84e414f8f317cc273b3226 Sep 10 02:33:14.186131 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.186111 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a8b7bf_b25c_49fc_8889_80653c50b02b.slice/crio-1c7d72d7fb18b58e97aeb55ac78e4bc2b9ef23283624b83dc4385208aaf30703 WatchSource:0}: Error finding container 1c7d72d7fb18b58e97aeb55ac78e4bc2b9ef23283624b83dc4385208aaf30703: Status 404 returned error can't find the container with id 1c7d72d7fb18b58e97aeb55ac78e4bc2b9ef23283624b83dc4385208aaf30703 Sep 10 02:33:14.192235 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:14.191611 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8458a415_6e5d_497d_8b39_bfa2aa005fea.slice/crio-cd2099f84a37b8d3ca09461a6b47e008080c4682384c9caf30781af290976da2 WatchSource:0}: Error finding container cd2099f84a37b8d3ca09461a6b47e008080c4682384c9caf30781af290976da2: Status 404 returned error can't find the container with id cd2099f84a37b8d3ca09461a6b47e008080c4682384c9caf30781af290976da2 Sep 10 02:33:14.197054 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.197016 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 02:28:12 +0000 UTC" deadline="2028-05-05 06:44:48.919062958 +0000 UTC" Sep 10 02:33:14.197054 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.197051 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14476h11m34.722013746s" Sep 10 02:33:14.234043 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.234017 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w7fk8" event={"ID":"1c19c39b-51ec-4ed1-8be8-f99fc93448a5","Type":"ContainerStarted","Data":"1b36c7748ef61d255e3bd48d2bbdb2a2778fbb108eba98acb2731165c4ddea1c"} Sep 10 02:33:14.235096 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.235076 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmppx" event={"ID":"8e1e758b-e536-410e-902e-864dd206dabd","Type":"ContainerStarted","Data":"bd66bf83c0cb6a9072e5a826468b60827cd30c4ca6cc239806e0b5762fdff37f"} Sep 10 02:33:14.236114 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.236096 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"d99d0a3731caf962af1b9c2dbe6f831430a31b4d9e823f3858aefd32a4f91571"} Sep 10 02:33:14.237053 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.237032 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" event={"ID":"8458a415-6e5d-497d-8b39-bfa2aa005fea","Type":"ContainerStarted","Data":"cd2099f84a37b8d3ca09461a6b47e008080c4682384c9caf30781af290976da2"} Sep 10 02:33:14.238344 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.238172 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" event={"ID":"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4","Type":"ContainerStarted","Data":"ac34d877e631e95fdf8fcfe8dc9cf8d16bbf1ef1fe9b9826d71492aac6299054"} Sep 10 02:33:14.239353 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.239323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4z6sw" event={"ID":"d280979a-b322-40d6-b312-4ae92bb42883","Type":"ContainerStarted","Data":"ccd54aa3c487b7a4119ae377f5d2b409888b114bd75b22dfda43ab9af73b491d"} Sep 10 02:33:14.240609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.240589 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvvrb" event={"ID":"55541690-c514-404a-8037-7bb94a66eaf5","Type":"ContainerStarted","Data":"fbd1917c04ae9cf247db99d73c83c1a25e178c5cb007f9def16ca637fd42405c"} Sep 10 02:33:14.241713 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.241693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerStarted","Data":"1c7d72d7fb18b58e97aeb55ac78e4bc2b9ef23283624b83dc4385208aaf30703"} Sep 10 02:33:14.247523 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.247502 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xc2b" event={"ID":"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9","Type":"ContainerStarted","Data":"6883f6613f6f61d62224baff22542df2967dc4df8f84e414f8f317cc273b3226"} Sep 10 02:33:14.784252 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.783614 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:14.784252 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.783761 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:14.784252 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.783822 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:16.78380529 +0000 UTC m=+6.049901673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:14.884394 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.884353 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:14.884554 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:14.884455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:14.884632 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.884595 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:33:14.884632 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.884613 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:33:14.884632 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.884627 2570 projected.go:196] Error preparing data for projected volume kube-api-access-gvd6b for pod openshift-network-diagnostics/network-check-target-9rzqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:14.884775 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.884684 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b podName:6c26c64a-148b-430a-b8a3-26ab98b62904 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:16.884666852 +0000 UTC m=+6.150763224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvd6b" (UniqueName: "kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b") pod "network-check-target-9rzqt" (UID: "6c26c64a-148b-430a-b8a3-26ab98b62904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:14.885124 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.885106 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:14.885214 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:14.885164 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret podName:3902b22c-0dda-4604-a7d8-439602edd83e nodeName:}" failed. No retries permitted until 2026-09-10 02:33:16.885148175 +0000 UTC m=+6.151244557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret") pod "global-pull-secret-syncer-sg8dz" (UID: "3902b22c-0dda-4604-a7d8-439602edd83e") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:15.233110 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:15.232159 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:15.233110 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:15.232280 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:15.233110 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:15.232918 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:15.233110 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:15.233016 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:15.236655 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:15.234527 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:15.236655 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:15.234617 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:15.264114 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:15.264076 2570 generic.go:356] "Generic (PLEG): container finished" podID="e4729ed97ef7a8c38fc61d9b5517096f" containerID="6dead6e0f25939ad55ad18fd7949f2c0ed927d1509e383e89f42a7698893fab0" exitCode=0 Sep 10 02:33:15.264271 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:15.264177 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" event={"ID":"e4729ed97ef7a8c38fc61d9b5517096f","Type":"ContainerDied","Data":"6dead6e0f25939ad55ad18fd7949f2c0ed927d1509e383e89f42a7698893fab0"} Sep 10 02:33:15.280183 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:15.280154 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" event={"ID":"728b04a33548859b274bbec2cc5b3fee","Type":"ContainerStarted","Data":"36d7e880277771387873bf8cf03a38dca319b84b32328207ca546084114d4e28"} Sep 10 02:33:16.349062 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:16.348950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" event={"ID":"e4729ed97ef7a8c38fc61d9b5517096f","Type":"ContainerStarted","Data":"488cda16b11a61fee826e24ce3afbcd928b4d036aed04a93a99abc6e8808b665"} Sep 10 02:33:16.363582 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:16.363522 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-228.ec2.internal" podStartSLOduration=4.363493475 podStartE2EDuration="4.363493475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:15.29624104 +0000 UTC m=+4.562337434" watchObservedRunningTime="2026-09-10 02:33:16.363493475 +0000 UTC m=+5.629589868" Sep 10 02:33:16.801907 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:16.801786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:16.802069 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.801941 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:16.802069 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.802009 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:20.801984621 +0000 UTC m=+10.068081007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:16.902472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:16.902527 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.902672 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.902733 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret podName:3902b22c-0dda-4604-a7d8-439602edd83e nodeName:}" failed. No retries permitted until 2026-09-10 02:33:20.902714746 +0000 UTC m=+10.168811130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret") pod "global-pull-secret-syncer-sg8dz" (UID: "3902b22c-0dda-4604-a7d8-439602edd83e") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.902961 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.902984 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.902996 2570 projected.go:196] Error preparing data for projected volume kube-api-access-gvd6b for pod openshift-network-diagnostics/network-check-target-9rzqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:16.903078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:16.903039 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b podName:6c26c64a-148b-430a-b8a3-26ab98b62904 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:20.903025802 +0000 UTC m=+10.169122185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvd6b" (UniqueName: "kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b") pod "network-check-target-9rzqt" (UID: "6c26c64a-148b-430a-b8a3-26ab98b62904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:17.222982 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:17.222452 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:17.222982 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:17.222580 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:17.222982 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:17.222451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:17.222982 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:17.222887 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:17.223293 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:17.222993 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:17.223293 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:17.223108 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:19.222430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:19.222394 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:19.222869 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:19.222529 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:19.222975 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:19.222912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:19.222975 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:19.222934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:19.223082 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:19.223025 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:19.223153 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:19.223127 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:20.835387 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:20.835352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:20.835960 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.835476 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:20.835960 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.835555 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:28.835534209 +0000 UTC m=+18.101630593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:20.935999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:20.936052 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.936188 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.936245 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret podName:3902b22c-0dda-4604-a7d8-439602edd83e nodeName:}" failed. No retries permitted until 2026-09-10 02:33:28.936228172 +0000 UTC m=+18.202324554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret") pod "global-pull-secret-syncer-sg8dz" (UID: "3902b22c-0dda-4604-a7d8-439602edd83e") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.936654 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.936675 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.936687 2570 projected.go:196] Error preparing data for projected volume kube-api-access-gvd6b for pod openshift-network-diagnostics/network-check-target-9rzqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:20.936762 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:20.936729 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b podName:6c26c64a-148b-430a-b8a3-26ab98b62904 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:28.936713908 +0000 UTC m=+18.202810291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvd6b" (UniqueName: "kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b") pod "network-check-target-9rzqt" (UID: "6c26c64a-148b-430a-b8a3-26ab98b62904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:21.222524 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:21.222448 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:21.222786 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:21.222494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:21.222894 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:21.222873 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:21.222959 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:21.222549 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:21.223094 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:21.223055 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:21.223477 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:21.223449 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:23.222181 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:23.221732 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:23.223464 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:23.222739 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:23.223464 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:23.223182 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:23.223464 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:23.223282 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:23.223464 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:23.223344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:23.223464 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:23.223426 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:25.223036 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:25.222399 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:25.223036 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:25.222410 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:25.223036 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:25.222541 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:25.223036 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:25.222541 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:25.223036 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:25.222652 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:25.223036 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:25.222740 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:27.221852 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:27.221807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:27.222310 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:27.221807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:27.222310 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:27.221943 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:27.222310 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:27.222058 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:27.222310 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:27.221807 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:27.222310 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:27.222143 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:28.894581 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:28.894550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:28.895152 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.894683 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:28.895152 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.894753 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:44.894730851 +0000 UTC m=+34.160827235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:28.995402 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:28.995368 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:28.995576 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:28.995418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:28.995576 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.995516 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:28.995683 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.995578 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret podName:3902b22c-0dda-4604-a7d8-439602edd83e nodeName:}" failed. No retries permitted until 2026-09-10 02:33:44.995557087 +0000 UTC m=+34.261653480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret") pod "global-pull-secret-syncer-sg8dz" (UID: "3902b22c-0dda-4604-a7d8-439602edd83e") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:28.995683 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.995518 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:33:28.995683 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.995641 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:33:28.995683 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.995662 2570 projected.go:196] Error preparing data for projected volume kube-api-access-gvd6b for pod openshift-network-diagnostics/network-check-target-9rzqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:28.995892 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:28.995743 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b podName:6c26c64a-148b-430a-b8a3-26ab98b62904 nodeName:}" failed. No retries permitted until 2026-09-10 02:33:44.995702637 +0000 UTC m=+34.261799019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvd6b" (UniqueName: "kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b") pod "network-check-target-9rzqt" (UID: "6c26c64a-148b-430a-b8a3-26ab98b62904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:29.222141 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:29.221964 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:29.222141 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:29.222111 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:29.222376 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:29.222170 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:29.222376 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:29.222262 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:29.222376 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:29.222303 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:29.222516 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:29.222382 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:31.223248 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:31.223211 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:31.223701 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:31.223301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:31.223701 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:31.223318 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:31.223701 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:31.223369 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:31.223701 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:31.223389 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:31.223701 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:31.223475 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:32.380729 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.380453 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" event={"ID":"8458a415-6e5d-497d-8b39-bfa2aa005fea","Type":"ContainerStarted","Data":"168fcfe2e7237e3e566e8e5eafbcd0d059d3f347a3a471e6451c83923a797e4d"} Sep 10 02:33:32.382061 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.382038 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" event={"ID":"9ec9e53c-6f18-4d3e-af62-2bcce14dc7d4","Type":"ContainerStarted","Data":"8a7851a05ca7dcd574b059348a6320c6761b8d7a0033d43b5504847b7422b7a9"} Sep 10 02:33:32.383502 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.383475 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gvvrb" event={"ID":"55541690-c514-404a-8037-7bb94a66eaf5","Type":"ContainerStarted","Data":"d75f4889bc68f9c02dd26d42d9bd128fcfaff55814e7cf3d122c7b10d0af46e7"} Sep 10 02:33:32.384907 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.384887 2570 generic.go:356] "Generic (PLEG): container finished" podID="85a8b7bf-b25c-49fc-8889-80653c50b02b" containerID="00b795810095a4cb529bc9fe42b9216c82381847943bc044cfea13637beb9443" exitCode=0 Sep 10 02:33:32.385005 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.384952 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerDied","Data":"00b795810095a4cb529bc9fe42b9216c82381847943bc044cfea13637beb9443"} Sep 10 02:33:32.386611 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.386593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xc2b" event={"ID":"14a6ac2d-65fb-4070-b0f0-979edbcf7aa9","Type":"ContainerStarted","Data":"9ce2571d7b05bccebd3b8676cd718406985ae27709860f0a3d591436cb94efc6"} Sep 10 02:33:32.388180 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.388159 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w7fk8" event={"ID":"1c19c39b-51ec-4ed1-8be8-f99fc93448a5","Type":"ContainerStarted","Data":"886d6c68562581525d6bd551bd6ec61b0f656bb57a21bd48358de0dc955ee7ec"} Sep 10 02:33:32.389525 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.389476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmppx" event={"ID":"8e1e758b-e536-410e-902e-864dd206dabd","Type":"ContainerStarted","Data":"a7fe3c5544ae1be1e0e139cf619bcfb21e806a4eff5df351a379bb75991f6c75"} Sep 10 02:33:32.392320 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.392302 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"9f4ffc2c5ec35fba9d432a92acf16701eb3ccafcc8b39b95ff9771c860f559ca"} Sep 10 02:33:32.392398 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.392353 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"d54c7a3731ff7bbc413457e11d2ac5a00e9436c804b0e85a1bb82d1c86ab1d9b"} Sep 10 02:33:32.392398 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.392383 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"0483dc5d22dfa9976d9851db9d963a2e7347ac918d668567751300567e63ac51"} Sep 10 02:33:32.392469 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.392411 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"dda904b58c82e49f370547a395b8c33575ec747ec95ab1e3198d09f3e57240e2"} Sep 10 02:33:32.392469 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.392438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"28f99acf306d8f3cd12328c271e69d24156f2c33ae72a83b0017fd694d561eb5"} Sep 10 02:33:32.392469 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.392467 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"257c12bd375769da3cb61994b52a14574985b338272e02b3b71f284a21628c04"} Sep 10 02:33:32.400495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.400457 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-228.ec2.internal" podStartSLOduration=20.400446181 podStartE2EDuration="20.400446181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:16.364467311 +0000 UTC m=+5.630563706" watchObservedRunningTime="2026-09-10 02:33:32.400446181 +0000 UTC m=+21.666542572" Sep 10 02:33:32.401093 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.401043 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m5zpn" podStartSLOduration=4.011130032 podStartE2EDuration="21.4010345s" totalImagesPullingTime="17.389904468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:32.400273954 +0000 UTC m=+21.666370346" watchObservedRunningTime="2026-09-10 02:33:32.4010345 +0000 UTC m=+21.667130908" Sep 10 02:33:32.414974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.414935 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bmppx" podStartSLOduration=7.947159873 podStartE2EDuration="21.414926057s" totalImagesPullingTime="13.467766184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:32.414587839 +0000 UTC m=+21.680684230" watchObservedRunningTime="2026-09-10 02:33:32.414926057 +0000 UTC m=+21.681022447" Sep 10 02:33:32.430231 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.430190 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gvvrb" podStartSLOduration=4.039899269 podStartE2EDuration="21.430178868s" totalImagesPullingTime="17.390279599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:32.429784718 +0000 UTC m=+21.695881108" watchObservedRunningTime="2026-09-10 02:33:32.430178868 +0000 UTC m=+21.696275258" Sep 10 02:33:32.444415 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.444376 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w7fk8" podStartSLOduration=7.978003456 podStartE2EDuration="21.444364967s" totalImagesPullingTime="13.466361511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:32.443929419 +0000 UTC m=+21.710025847" watchObservedRunningTime="2026-09-10 02:33:32.444364967 +0000 UTC m=+21.710461359" Sep 10 02:33:32.465635 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:32.465594 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8xc2b" podStartSLOduration=4.024580133 podStartE2EDuration="21.46558296s" totalImagesPullingTime="17.441002827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:32.465288059 +0000 UTC m=+21.731384451" watchObservedRunningTime="2026-09-10 02:33:32.46558296 +0000 UTC m=+21.731679352" Sep 10 02:33:33.222509 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:33.222311 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:33.222675 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:33.222323 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:33.222675 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:33.222593 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:33.222802 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:33.222391 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:33.222802 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:33.222705 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:33.222927 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:33.222813 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:33.343323 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:33.343291 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 02:33:33.396793 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:33.396756 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" event={"ID":"8458a415-6e5d-497d-8b39-bfa2aa005fea","Type":"ContainerStarted","Data":"69e9b74b7fb30098c04085a511d41b4b7b1a8e123a08a4bceeef21494d293997"} Sep 10 02:33:33.398586 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:33.398533 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4z6sw" event={"ID":"d280979a-b322-40d6-b312-4ae92bb42883","Type":"ContainerStarted","Data":"64dd01d06e8e9e210ba00d30af71f0a22383ec326ced6708ef0ce2f3065ea726"} Sep 10 02:33:33.414592 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:33.414552 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4z6sw" podStartSLOduration=8.944803775 podStartE2EDuration="22.41453856s" totalImagesPullingTime="13.469734785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:33.4142332 +0000 UTC m=+22.680329592" watchObservedRunningTime="2026-09-10 02:33:33.41453856 +0000 UTC m=+22.680634992" Sep 10 02:33:34.225433 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:34.225308 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T02:33:33.343318127Z","UUID":"73d6c6fa-4f4a-46db-9f2c-0791d68ba0c0","Handler":null,"Name":"","Endpoint":""} Sep 10 02:33:34.227200 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:34.227177 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 02:33:34.227335 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:34.227214 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 02:33:34.406940 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:34.406876 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"132cd1c4b5baad79e06a4635ef07e9885f25092235842e083e87c2faa64e8944"} Sep 10 02:33:35.007959 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:35.007931 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:35.008524 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:35.008503 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:35.222173 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:35.222140 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:35.222173 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:35.222172 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:35.222398 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:35.222148 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:35.222398 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:35.222273 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:35.222398 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:35.222352 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:35.222517 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:35.222442 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:35.412003 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:35.411907 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" event={"ID":"8458a415-6e5d-497d-8b39-bfa2aa005fea","Type":"ContainerStarted","Data":"92809fe61d956fcc7cd3062646c313ed94ce259268c0243ccb0c472e9220ea44"} Sep 10 02:33:35.428894 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:35.428841 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtnkj" podStartSLOduration=4.587674462 podStartE2EDuration="24.428803921s" totalImagesPullingTime="19.841129459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:35.428399605 +0000 UTC m=+24.694495995" watchObservedRunningTime="2026-09-10 02:33:35.428803921 +0000 UTC m=+24.694900313" Sep 10 02:33:37.221493 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:37.221456 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:37.221942 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:37.221456 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:37.221942 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:37.221578 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:37.221942 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:37.221572 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:37.221942 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:37.221678 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:37.221942 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:37.221756 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:38.420326 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:38.420088 2570 generic.go:356] "Generic (PLEG): container finished" podID="85a8b7bf-b25c-49fc-8889-80653c50b02b" containerID="438d557f0e7bcaa45568a820df0a10682a713edaaea9a91beeab403dd2e98dee" exitCode=0 Sep 10 02:33:38.420945 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:38.420175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerDied","Data":"438d557f0e7bcaa45568a820df0a10682a713edaaea9a91beeab403dd2e98dee"} Sep 10 02:33:38.424295 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:38.424273 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" event={"ID":"ca08812a-7a23-4716-8d22-e116d09a39a4","Type":"ContainerStarted","Data":"f9035618dbb9497a5652632551a64d14c2bd0fbc8006d49a3a2e4e34373719ba"} Sep 10 02:33:38.424513 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:38.424499 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:38.439458 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:38.439441 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:38.465739 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:38.465699 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" podStartSLOduration=10.015308916 podStartE2EDuration="27.465685102s" totalImagesPullingTime="17.450376186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:38.46454679 +0000 UTC m=+27.730643181" watchObservedRunningTime="2026-09-10 02:33:38.465685102 +0000 UTC m=+27.731781476" Sep 10 02:33:39.221627 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.221596 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:39.221848 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.221600 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:39.221848 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:39.221723 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:39.221848 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.221736 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:39.221848 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:39.221813 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:39.222076 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:39.221907 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:39.427381 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.427348 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:39.428006 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.427444 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:39.447088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.447056 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:33:39.776940 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.776880 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9rzqt"] Sep 10 02:33:39.777051 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.776980 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:39.777094 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:39.777050 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:39.780415 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.780389 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6hhxs"] Sep 10 02:33:39.780523 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.780506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:39.780642 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:39.780620 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:39.781244 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.781030 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sg8dz"] Sep 10 02:33:39.781244 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:39.781161 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:39.781429 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:39.781256 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:40.431122 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:40.431084 2570 generic.go:356] "Generic (PLEG): container finished" podID="85a8b7bf-b25c-49fc-8889-80653c50b02b" containerID="f9418e7cc8a88b3e1061b5663ac6f5607a9c4199ed7d79f1b05b3dd6d3674568" exitCode=0 Sep 10 02:33:40.431566 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:40.431142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerDied","Data":"f9418e7cc8a88b3e1061b5663ac6f5607a9c4199ed7d79f1b05b3dd6d3674568"} Sep 10 02:33:41.222528 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:41.222331 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:41.222726 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:41.222613 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:42.221662 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:42.221584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:42.222078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:42.221684 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:42.222078 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:42.221750 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:42.222078 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:42.221883 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:42.437892 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:42.437857 2570 generic.go:356] "Generic (PLEG): container finished" podID="85a8b7bf-b25c-49fc-8889-80653c50b02b" containerID="90e7c43fc3cf2f4017302306cae890d941bccb729cbe999288de0b3698b15d42" exitCode=0 Sep 10 02:33:42.438057 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:42.437911 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerDied","Data":"90e7c43fc3cf2f4017302306cae890d941bccb729cbe999288de0b3698b15d42"} Sep 10 02:33:43.222192 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:43.222164 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:43.222603 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:43.222276 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:43.466565 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:43.466525 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:43.466742 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:43.466676 2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 02:33:43.467355 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:43.467330 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bmppx" Sep 10 02:33:44.221598 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:44.221564 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:44.221598 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:44.221600 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:44.221855 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:44.221684 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhxs" podUID="253d3186-27d1-4b7e-b48a-303871be8896" Sep 10 02:33:44.221855 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:44.221821 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sg8dz" podUID="3902b22c-0dda-4604-a7d8-439602edd83e" Sep 10 02:33:44.907626 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:44.907547 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:44.908104 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:44.907677 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:44.908104 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:44.907742 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:16.9077246 +0000 UTC m=+66.173820983 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 02:33:45.008571 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.008530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:45.008753 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.008592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:45.008753 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.008683 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:45.008886 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.008754 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret podName:3902b22c-0dda-4604-a7d8-439602edd83e nodeName:}" failed. No retries permitted until 2026-09-10 02:34:17.00873835 +0000 UTC m=+66.274834723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret") pod "global-pull-secret-syncer-sg8dz" (UID: "3902b22c-0dda-4604-a7d8-439602edd83e") : object "kube-system"/"original-pull-secret" not registered Sep 10 02:33:45.008886 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.008686 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 02:33:45.008886 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.008788 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 02:33:45.008886 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.008800 2570 projected.go:196] Error preparing data for projected volume kube-api-access-gvd6b for pod openshift-network-diagnostics/network-check-target-9rzqt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:45.008886 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.008854 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b podName:6c26c64a-148b-430a-b8a3-26ab98b62904 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:17.008842012 +0000 UTC m=+66.274938395 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvd6b" (UniqueName: "kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b") pod "network-check-target-9rzqt" (UID: "6c26c64a-148b-430a-b8a3-26ab98b62904") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 02:33:45.222426 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.222397 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:45.222576 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.222530 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rzqt" podUID="6c26c64a-148b-430a-b8a3-26ab98b62904" Sep 10 02:33:45.541974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.541904 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-228.ec2.internal" event="NodeReady" Sep 10 02:33:45.542120 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.542058 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 02:33:45.580700 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.580665 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b47995f-scq9k"] Sep 10 02:33:45.584021 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.584000 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.586475 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.586454 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pb55z\"" Sep 10 02:33:45.586576 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.586483 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 02:33:45.586576 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.586513 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 02:33:45.586650 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.586454 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 02:33:45.593715 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.593693 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 02:33:45.595550 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.595530 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b47995f-scq9k"] Sep 10 02:33:45.599064 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.599011 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v8xtg"] Sep 10 02:33:45.602016 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.601998 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-whf7d"] Sep 10 02:33:45.602149 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.602134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.604869 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.604582 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4fcfm\"" Sep 10 02:33:45.604869 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.604650 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 02:33:45.604869 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.604718 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:45.604869 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.604777 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 02:33:45.607430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.607408 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 02:33:45.607524 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.607474 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 02:33:45.607817 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.607651 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 02:33:45.607924 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.607872 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v9q4l\"" Sep 10 02:33:45.616957 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.616940 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whf7d"] Sep 10 02:33:45.617726 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.617704 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v8xtg"] Sep 10 02:33:45.713426 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713393 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d322117c-a2b5-4598-8c62-31ab1911f3cd-tmp-dir\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.713608 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713437 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-installation-pull-secrets\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.713608 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d322117c-a2b5-4598-8c62-31ab1911f3cd-config-volume\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.713608 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-image-registry-private-configuration\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.713608 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc7a6fc4-786a-4e2e-84d9-283835e132ed-ca-trust-extracted\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713676 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4mj\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-kube-api-access-ql4mj\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713718 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2w94\" (UniqueName: \"kubernetes.io/projected/9d0bb233-da91-40b6-963b-80352b7f96ee-kube-api-access-w2w94\") pod \"ingress-canary-whf7d\" (UID: \"9d0bb233-da91-40b6-963b-80352b7f96ee\") " pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713769 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d0bb233-da91-40b6-963b-80352b7f96ee-cert\") pod \"ingress-canary-whf7d\" (UID: \"9d0bb233-da91-40b6-963b-80352b7f96ee\") " pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713788 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6psg2\" (UniqueName: \"kubernetes.io/projected/d322117c-a2b5-4598-8c62-31ab1911f3cd-kube-api-access-6psg2\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-certificates\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.713843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713821 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-trusted-ca\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.714246 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.713850 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-bound-sa-token\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814311 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-bound-sa-token\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814311 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d322117c-a2b5-4598-8c62-31ab1911f3cd-tmp-dir\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.814525 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-installation-pull-secrets\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814525 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814478 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d322117c-a2b5-4598-8c62-31ab1911f3cd-config-volume\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.814525 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-image-registry-private-configuration\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814571 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc7a6fc4-786a-4e2e-84d9-283835e132ed-ca-trust-extracted\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.814679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814600 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d322117c-a2b5-4598-8c62-31ab1911f3cd-tmp-dir\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.814679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814623 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4mj\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-kube-api-access-ql4mj\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2w94\" (UniqueName: \"kubernetes.io/projected/9d0bb233-da91-40b6-963b-80352b7f96ee-kube-api-access-w2w94\") pod \"ingress-canary-whf7d\" (UID: \"9d0bb233-da91-40b6-963b-80352b7f96ee\") " pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:45.814985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814704 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d0bb233-da91-40b6-963b-80352b7f96ee-cert\") pod \"ingress-canary-whf7d\" (UID: \"9d0bb233-da91-40b6-963b-80352b7f96ee\") " pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:45.814985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6psg2\" (UniqueName: \"kubernetes.io/projected/d322117c-a2b5-4598-8c62-31ab1911f3cd-kube-api-access-6psg2\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.814985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814755 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-certificates\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.814780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-trusted-ca\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.814985 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.814945 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:33:45.815213 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.815042 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:33:45.815213 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.815058 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b47995f-scq9k: secret "image-registry-tls" not found Sep 10 02:33:45.815213 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.815062 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls podName:d322117c-a2b5-4598-8c62-31ab1911f3cd nodeName:}" failed. No retries permitted until 2026-09-10 02:33:46.315039396 +0000 UTC m=+35.581135780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls") pod "dns-default-v8xtg" (UID: "d322117c-a2b5-4598-8c62-31ab1911f3cd") : secret "dns-default-metrics-tls" not found Sep 10 02:33:45.815213 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:45.815103 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls podName:dc7a6fc4-786a-4e2e-84d9-283835e132ed nodeName:}" failed. No retries permitted until 2026-09-10 02:33:46.315086974 +0000 UTC m=+35.581183363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls") pod "image-registry-5b47995f-scq9k" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed") : secret "image-registry-tls" not found Sep 10 02:33:45.817235 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.817066 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc7a6fc4-786a-4e2e-84d9-283835e132ed-ca-trust-extracted\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.817597 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.817378 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-certificates\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.817780 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.817757 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-trusted-ca\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.818011 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.817987 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d322117c-a2b5-4598-8c62-31ab1911f3cd-config-volume\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.821374 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.821336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d0bb233-da91-40b6-963b-80352b7f96ee-cert\") pod \"ingress-canary-whf7d\" (UID: \"9d0bb233-da91-40b6-963b-80352b7f96ee\") " pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:45.821374 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.821356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-image-registry-private-configuration\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.822387 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.822328 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-installation-pull-secrets\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.823643 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.823619 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4mj\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-kube-api-access-ql4mj\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.823758 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.823732 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2w94\" (UniqueName: \"kubernetes.io/projected/9d0bb233-da91-40b6-963b-80352b7f96ee-kube-api-access-w2w94\") pod \"ingress-canary-whf7d\" (UID: \"9d0bb233-da91-40b6-963b-80352b7f96ee\") " pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:45.824639 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.824615 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-bound-sa-token\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:45.825074 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.825043 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6psg2\" (UniqueName: \"kubernetes.io/projected/d322117c-a2b5-4598-8c62-31ab1911f3cd-kube-api-access-6psg2\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:45.919117 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:45.919082 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whf7d" Sep 10 02:33:46.222093 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:46.222057 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:33:46.222270 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:46.222098 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:33:46.224845 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:46.224806 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 02:33:46.224845 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:46.224811 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwtct\"" Sep 10 02:33:46.225025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:46.224877 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 02:33:46.319294 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:46.319264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:46.319472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:46.319313 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:46.319472 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:46.319418 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:33:46.319472 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:46.319438 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b47995f-scq9k: secret "image-registry-tls" not found Sep 10 02:33:46.319472 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:46.319460 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:33:46.319681 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:46.319505 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls podName:dc7a6fc4-786a-4e2e-84d9-283835e132ed nodeName:}" failed. No retries permitted until 2026-09-10 02:33:47.319483487 +0000 UTC m=+36.585579857 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls") pod "image-registry-5b47995f-scq9k" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed") : secret "image-registry-tls" not found Sep 10 02:33:46.319681 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:46.319523 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls podName:d322117c-a2b5-4598-8c62-31ab1911f3cd nodeName:}" failed. No retries permitted until 2026-09-10 02:33:47.319514207 +0000 UTC m=+36.585610576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls") pod "dns-default-v8xtg" (UID: "d322117c-a2b5-4598-8c62-31ab1911f3cd") : secret "dns-default-metrics-tls" not found Sep 10 02:33:47.222081 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.222021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:33:47.224688 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.224666 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 02:33:47.224835 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.224798 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 02:33:47.224988 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.224966 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tcpg8\"" Sep 10 02:33:47.245025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.244979 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whf7d"] Sep 10 02:33:47.248851 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:33:47.248817 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0bb233_da91_40b6_963b_80352b7f96ee.slice/crio-c97906478fcceb56f84d4c4974a7c90a0dc6f7e38bdf9a3a9e47b2ac3be3806a WatchSource:0}: Error finding container c97906478fcceb56f84d4c4974a7c90a0dc6f7e38bdf9a3a9e47b2ac3be3806a: Status 404 returned error can't find the container with id c97906478fcceb56f84d4c4974a7c90a0dc6f7e38bdf9a3a9e47b2ac3be3806a Sep 10 02:33:47.326434 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.326402 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:47.326563 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.326464 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:47.326630 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:47.326600 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:33:47.326630 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:47.326623 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b47995f-scq9k: secret "image-registry-tls" not found Sep 10 02:33:47.326717 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:47.326633 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:33:47.326717 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:47.326699 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls podName:dc7a6fc4-786a-4e2e-84d9-283835e132ed nodeName:}" failed. No retries permitted until 2026-09-10 02:33:49.326677031 +0000 UTC m=+38.592773409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls") pod "image-registry-5b47995f-scq9k" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed") : secret "image-registry-tls" not found Sep 10 02:33:47.326793 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:47.326720 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls podName:d322117c-a2b5-4598-8c62-31ab1911f3cd nodeName:}" failed. No retries permitted until 2026-09-10 02:33:49.326710219 +0000 UTC m=+38.592806592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls") pod "dns-default-v8xtg" (UID: "d322117c-a2b5-4598-8c62-31ab1911f3cd") : secret "dns-default-metrics-tls" not found Sep 10 02:33:47.451138 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.451097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whf7d" event={"ID":"9d0bb233-da91-40b6-963b-80352b7f96ee","Type":"ContainerStarted","Data":"c97906478fcceb56f84d4c4974a7c90a0dc6f7e38bdf9a3a9e47b2ac3be3806a"} Sep 10 02:33:47.454034 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.454008 2570 generic.go:356] "Generic (PLEG): container finished" podID="85a8b7bf-b25c-49fc-8889-80653c50b02b" containerID="47a0191e448afde957a3590fdefa8d9469c5bcee926ab54974bcd62c44192f0a" exitCode=0 Sep 10 02:33:47.454158 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:47.454060 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerDied","Data":"47a0191e448afde957a3590fdefa8d9469c5bcee926ab54974bcd62c44192f0a"} Sep 10 02:33:48.460596 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:48.460391 2570 generic.go:356] "Generic (PLEG): container finished" podID="85a8b7bf-b25c-49fc-8889-80653c50b02b" containerID="4ae5aaf3d51e949073b077ab97c75072949006fed7f1b497e40bc1a70b65a03d" exitCode=0 Sep 10 02:33:48.461070 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:48.460509 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerDied","Data":"4ae5aaf3d51e949073b077ab97c75072949006fed7f1b497e40bc1a70b65a03d"} Sep 10 02:33:48.946713 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:48.946680 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:49.040537 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:49.040501 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:49.340324 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:49.340291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:49.340461 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:49.340334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:49.340461 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:49.340440 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:33:49.340519 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:49.340455 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:33:49.340519 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:49.340473 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b47995f-scq9k: secret "image-registry-tls" not found Sep 10 02:33:49.340519 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:49.340489 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls podName:d322117c-a2b5-4598-8c62-31ab1911f3cd nodeName:}" failed. No retries permitted until 2026-09-10 02:33:53.340477854 +0000 UTC m=+42.606574223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls") pod "dns-default-v8xtg" (UID: "d322117c-a2b5-4598-8c62-31ab1911f3cd") : secret "dns-default-metrics-tls" not found Sep 10 02:33:49.340630 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:49.340520 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls podName:dc7a6fc4-786a-4e2e-84d9-283835e132ed nodeName:}" failed. No retries permitted until 2026-09-10 02:33:53.340506768 +0000 UTC m=+42.606603150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls") pod "image-registry-5b47995f-scq9k" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed") : secret "image-registry-tls" not found Sep 10 02:33:49.466144 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:49.466109 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-87cr5" event={"ID":"85a8b7bf-b25c-49fc-8889-80653c50b02b","Type":"ContainerStarted","Data":"6294f420911b8fa24fc50bba6f00523800c036e012d3dee39cb519563d3b21de"} Sep 10 02:33:49.489956 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:49.489910 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-87cr5" podStartSLOduration=8.737913479 podStartE2EDuration="38.489896054s" totalImagesPullingTime="29.751982575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:49.488264301 +0000 UTC m=+38.754360691" watchObservedRunningTime="2026-09-10 02:33:49.489896054 +0000 UTC m=+38.755992450" Sep 10 02:33:49.609640 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:49.609559 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:49.669655 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:49.669629 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:51.471938 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:51.471899 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whf7d" event={"ID":"9d0bb233-da91-40b6-963b-80352b7f96ee","Type":"ContainerStarted","Data":"cfbcbfc482f2110c8cfcce243646d419d1bbaf121417cb2fdefb7029db6c06a9"} Sep 10 02:33:51.487161 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:51.487112 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-whf7d" podStartSLOduration=2.9038747320000002 podStartE2EDuration="6.487093533s" totalImagesPullingTime="3.583218801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:33:51.486329813 +0000 UTC m=+40.752426216" watchObservedRunningTime="2026-09-10 02:33:51.487093533 +0000 UTC m=+40.753189923" Sep 10 02:33:53.370291 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:53.370260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:33:53.370648 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:53.370298 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:33:53.370648 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:53.370394 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:33:53.370648 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:53.370400 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:33:53.370648 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:53.370415 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b47995f-scq9k: secret "image-registry-tls" not found Sep 10 02:33:53.370648 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:53.370439 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls podName:d322117c-a2b5-4598-8c62-31ab1911f3cd nodeName:}" failed. No retries permitted until 2026-09-10 02:34:01.370425695 +0000 UTC m=+50.636522064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls") pod "dns-default-v8xtg" (UID: "d322117c-a2b5-4598-8c62-31ab1911f3cd") : secret "dns-default-metrics-tls" not found Sep 10 02:33:53.370648 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:33:53.370451 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls podName:dc7a6fc4-786a-4e2e-84d9-283835e132ed nodeName:}" failed. No retries permitted until 2026-09-10 02:34:01.370445308 +0000 UTC m=+50.636541677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls") pod "image-registry-5b47995f-scq9k" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed") : secret "image-registry-tls" not found Sep 10 02:33:54.432033 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:54.431999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:54.509291 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:54.509261 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:54.592355 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:54.592322 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:54.665159 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:54.665115 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:54.730381 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:54.730286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:55.454072 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:55.454035 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:55.838379 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:55.838302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:33:55.895211 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:33:55.895177 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:01.428007 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:01.427960 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:34:01.428007 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:01.428009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:34:01.428456 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:01.428117 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:34:01.428456 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:01.428125 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:34:01.428456 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:01.428141 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b47995f-scq9k: secret "image-registry-tls" not found Sep 10 02:34:01.428456 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:01.428184 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls podName:d322117c-a2b5-4598-8c62-31ab1911f3cd nodeName:}" failed. No retries permitted until 2026-09-10 02:34:17.428165616 +0000 UTC m=+66.694262004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls") pod "dns-default-v8xtg" (UID: "d322117c-a2b5-4598-8c62-31ab1911f3cd") : secret "dns-default-metrics-tls" not found Sep 10 02:34:01.428456 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:01.428199 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls podName:dc7a6fc4-786a-4e2e-84d9-283835e132ed nodeName:}" failed. No retries permitted until 2026-09-10 02:34:17.428193073 +0000 UTC m=+66.694289442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls") pod "image-registry-5b47995f-scq9k" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed") : secret "image-registry-tls" not found Sep 10 02:34:02.060057 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.060011 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p"] Sep 10 02:34:02.091224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.091198 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl"] Sep 10 02:34:02.091368 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.091319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" Sep 10 02:34:02.094175 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.094152 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.094175 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.094175 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.094375 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.094225 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zmlm5\"" Sep 10 02:34:02.114978 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.114954 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p"] Sep 10 02:34:02.115133 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.114986 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl"] Sep 10 02:34:02.115133 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.115080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:02.117637 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.117617 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.117736 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.117651 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 02:34:02.117736 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.117728 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mmd8f\"" Sep 10 02:34:02.117937 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.117728 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.157593 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.157570 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-djzb9"] Sep 10 02:34:02.172756 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.172734 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-djzb9"] Sep 10 02:34:02.172873 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.172837 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" Sep 10 02:34:02.175794 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.175775 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sbg7h\"" Sep 10 02:34:02.181509 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.181489 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg"] Sep 10 02:34:02.197439 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.197418 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp"] Sep 10 02:34:02.197571 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.197545 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.200062 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.200043 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 02:34:02.200158 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.200059 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 02:34:02.200229 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.200059 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.200229 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.200077 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.200891 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.200868 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mh5fc\"" Sep 10 02:34:02.212276 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.212255 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-bk7gd"] Sep 10 02:34:02.212405 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.212388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.216636 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.216618 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 02:34:02.216736 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.216627 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.216736 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.216661 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.218113 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.218094 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 02:34:02.218293 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.218269 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8vbjv\"" Sep 10 02:34:02.230724 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.230704 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.230724 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.230721 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg"] Sep 10 02:34:02.230934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.230780 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-bk7gd"] Sep 10 02:34:02.230934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.230798 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp"] Sep 10 02:34:02.233063 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.233036 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:02.233157 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.233148 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d948k\" (UniqueName: \"kubernetes.io/projected/301963d9-6567-47f5-9484-e9525db81b46-kube-api-access-d948k\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:02.233202 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.233178 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25z4x\" (UniqueName: \"kubernetes.io/projected/41df8304-9f36-45be-a5c7-696eb836d8e9-kube-api-access-25z4x\") pod \"volume-data-source-validator-695c895bd6-4c45p\" (UID: \"41df8304-9f36-45be-a5c7-696eb836d8e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" Sep 10 02:34:02.234068 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.234051 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zf9xl\"" Sep 10 02:34:02.234068 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.234067 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 02:34:02.234204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.234116 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.234204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.234112 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 02:34:02.234365 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.234352 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.238905 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.238884 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 02:34:02.266569 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.266541 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c"] Sep 10 02:34:02.284104 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.284079 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-xr5ft"] Sep 10 02:34:02.284243 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.284221 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.286847 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.286815 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.286847 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.286820 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.287161 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.287145 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 02:34:02.287245 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.287148 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-575m6\"" Sep 10 02:34:02.287372 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.287356 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 02:34:02.300503 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.300482 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6fb5dd5bbd-khhbd"] Sep 10 02:34:02.300598 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.300486 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.303480 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.303461 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 02:34:02.303576 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.303501 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.303576 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.303508 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 02:34:02.303700 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.303587 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.304662 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.304644 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9khgk\"" Sep 10 02:34:02.308451 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.308433 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 02:34:02.323501 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.323450 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c"] Sep 10 02:34:02.323501 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.323477 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-xr5ft"] Sep 10 02:34:02.323501 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.323491 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fb5dd5bbd-khhbd"] Sep 10 02:34:02.323645 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.323557 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.325996 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.325976 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 02:34:02.326094 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.326001 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 02:34:02.326094 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.325980 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 02:34:02.326094 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.326005 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 02:34:02.326242 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.326210 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 02:34:02.326422 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.326404 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 02:34:02.326498 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.326447 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-px9cc\"" Sep 10 02:34:02.333764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.333746 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:02.333920 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.333778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d948k\" (UniqueName: \"kubernetes.io/projected/301963d9-6567-47f5-9484-e9525db81b46-kube-api-access-d948k\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:02.333920 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.333808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/92db0478-ff93-4fb6-a696-0885419be325-snapshots\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.333920 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.333846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24a99e6f-fe48-48fc-a340-09ae9d1728fc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.333920 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.333877 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm7fx\" (UniqueName: \"kubernetes.io/projected/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-kube-api-access-rm7fx\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.333920 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.333895 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.333960 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls podName:301963d9-6567-47f5-9484-e9525db81b46 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:02.83393625 +0000 UTC m=+52.100032621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lnprl" (UID: "301963d9-6567-47f5-9484-e9525db81b46") : secret "samples-operator-tls" not found Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.333895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25z4x\" (UniqueName: \"kubernetes.io/projected/41df8304-9f36-45be-a5c7-696eb836d8e9-kube-api-access-25z4x\") pod \"volume-data-source-validator-695c895bd6-4c45p\" (UID: \"41df8304-9f36-45be-a5c7-696eb836d8e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92db0478-ff93-4fb6-a696-0885419be325-tmp\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92db0478-ff93-4fb6-a696-0885419be325-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkg9n\" (UniqueName: \"kubernetes.io/projected/24a99e6f-fe48-48fc-a340-09ae9d1728fc-kube-api-access-dkg9n\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.334168 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6szp\" (UniqueName: \"kubernetes.io/projected/92db0478-ff93-4fb6-a696-0885419be325-kube-api-access-l6szp\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.334445 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92db0478-ff93-4fb6-a696-0885419be325-serving-cert\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.334445 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334222 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92db0478-ff93-4fb6-a696-0885419be325-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.334445 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334243 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24a99e6f-fe48-48fc-a340-09ae9d1728fc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.334445 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.334271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwqx\" (UniqueName: \"kubernetes.io/projected/8516628f-ff2b-402d-985f-354faf0f1478-kube-api-access-dkwqx\") pod \"network-check-source-685456655f-djzb9\" (UID: \"8516628f-ff2b-402d-985f-354faf0f1478\") " pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" Sep 10 02:34:02.346714 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.346692 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25z4x\" (UniqueName: \"kubernetes.io/projected/41df8304-9f36-45be-a5c7-696eb836d8e9-kube-api-access-25z4x\") pod \"volume-data-source-validator-695c895bd6-4c45p\" (UID: \"41df8304-9f36-45be-a5c7-696eb836d8e9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" Sep 10 02:34:02.346870 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.346850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d948k\" (UniqueName: \"kubernetes.io/projected/301963d9-6567-47f5-9484-e9525db81b46-kube-api-access-d948k\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:02.399980 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.399951 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" Sep 10 02:34:02.410114 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.410084 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:02.435080 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435053 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6szp\" (UniqueName: \"kubernetes.io/projected/92db0478-ff93-4fb6-a696-0885419be325-kube-api-access-l6szp\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.435435 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92db0478-ff93-4fb6-a696-0885419be325-serving-cert\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.435435 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92db0478-ff93-4fb6-a696-0885419be325-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.435435 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24a99e6f-fe48-48fc-a340-09ae9d1728fc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.435435 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwqx\" (UniqueName: \"kubernetes.io/projected/8516628f-ff2b-402d-985f-354faf0f1478-kube-api-access-dkwqx\") pod \"network-check-source-685456655f-djzb9\" (UID: \"8516628f-ff2b-402d-985f-354faf0f1478\") " pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" Sep 10 02:34:02.435435 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-stats-auth\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.435662 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2400c62-09d7-4e9a-9640-85ce68a09fb4-serving-cert\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.435662 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/92db0478-ff93-4fb6-a696-0885419be325-snapshots\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.435662 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2400c62-09d7-4e9a-9640-85ce68a09fb4-trusted-ca\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.435662 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435564 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24a99e6f-fe48-48fc-a340-09ae9d1728fc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.435821 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435714 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ea64165-5263-4e39-b47c-d69201ac3a2b-serving-cert\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.435821 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435747 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm7fx\" (UniqueName: \"kubernetes.io/projected/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-kube-api-access-rm7fx\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.435821 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247v6\" (UniqueName: \"kubernetes.io/projected/a2400c62-09d7-4e9a-9640-85ce68a09fb4-kube-api-access-247v6\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.435821 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.436088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435871 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.436088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92db0478-ff93-4fb6-a696-0885419be325-tmp\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.436088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435929 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.436088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.435956 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9z9\" (UniqueName: \"kubernetes.io/projected/d22aa7ab-c491-460d-8415-d452a6a64976-kube-api-access-cp9z9\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.436088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.436088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24a99e6f-fe48-48fc-a340-09ae9d1728fc-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.436088 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436051 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2400c62-09d7-4e9a-9640-85ce68a09fb4-config\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.436404 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436140 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92db0478-ff93-4fb6-a696-0885419be325-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.436404 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.436159 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:02.436404 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436173 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-default-certificate\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.436404 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkg9n\" (UniqueName: \"kubernetes.io/projected/24a99e6f-fe48-48fc-a340-09ae9d1728fc-kube-api-access-dkg9n\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.436404 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ea64165-5263-4e39-b47c-d69201ac3a2b-config\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.436404 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.436261 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls podName:ffbf8b54-3714-486c-bc13-4eeb9cd28d26 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:02.936241583 +0000 UTC m=+52.202337973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-frrmp" (UID: "ffbf8b54-3714-486c-bc13-4eeb9cd28d26") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:02.436404 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf7mw\" (UniqueName: \"kubernetes.io/projected/6ea64165-5263-4e39-b47c-d69201ac3a2b-kube-api-access-mf7mw\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.436666 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.436608 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.437762 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.437737 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24a99e6f-fe48-48fc-a340-09ae9d1728fc-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.443490 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.443469 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwqx\" (UniqueName: \"kubernetes.io/projected/8516628f-ff2b-402d-985f-354faf0f1478-kube-api-access-dkwqx\") pod \"network-check-source-685456655f-djzb9\" (UID: \"8516628f-ff2b-402d-985f-354faf0f1478\") " pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" Sep 10 02:34:02.445086 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.445058 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm7fx\" (UniqueName: \"kubernetes.io/projected/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-kube-api-access-rm7fx\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.445148 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.445085 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkg9n\" (UniqueName: \"kubernetes.io/projected/24a99e6f-fe48-48fc-a340-09ae9d1728fc-kube-api-access-dkg9n\") pod \"kube-storage-version-migrator-operator-8d7dd4675-szhxg\" (UID: \"24a99e6f-fe48-48fc-a340-09ae9d1728fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.449122 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.449097 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/92db0478-ff93-4fb6-a696-0885419be325-serving-cert\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.449122 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.449116 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92db0478-ff93-4fb6-a696-0885419be325-tmp\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.449378 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.449356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92db0478-ff93-4fb6-a696-0885419be325-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.449457 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.449385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/92db0478-ff93-4fb6-a696-0885419be325-snapshots\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.449530 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.449514 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92db0478-ff93-4fb6-a696-0885419be325-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.451125 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.451105 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6szp\" (UniqueName: \"kubernetes.io/projected/92db0478-ff93-4fb6-a696-0885419be325-kube-api-access-l6szp\") pod \"insights-operator-76bcd6ccf5-bk7gd\" (UID: \"92db0478-ff93-4fb6-a696-0885419be325\") " pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.481092 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.481067 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" Sep 10 02:34:02.509352 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.509327 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" Sep 10 02:34:02.512123 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.512097 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:02.536705 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.536675 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9z9\" (UniqueName: \"kubernetes.io/projected/d22aa7ab-c491-460d-8415-d452a6a64976-kube-api-access-cp9z9\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.536804 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.536743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2400c62-09d7-4e9a-9640-85ce68a09fb4-config\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.536804 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.536776 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-default-certificate\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.536895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.536804 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ea64165-5263-4e39-b47c-d69201ac3a2b-config\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.536895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.536849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf7mw\" (UniqueName: \"kubernetes.io/projected/6ea64165-5263-4e39-b47c-d69201ac3a2b-kube-api-access-mf7mw\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.536956 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.536903 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-stats-auth\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.537011 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.536994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2400c62-09d7-4e9a-9640-85ce68a09fb4-serving-cert\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.537503 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.537438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2400c62-09d7-4e9a-9640-85ce68a09fb4-trusted-ca\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.537609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.537505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ea64165-5263-4e39-b47c-d69201ac3a2b-serving-cert\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.537609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.537545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247v6\" (UniqueName: \"kubernetes.io/projected/a2400c62-09d7-4e9a-9640-85ce68a09fb4-kube-api-access-247v6\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.537609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.537569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.537609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.537603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.537985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.537626 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ea64165-5263-4e39-b47c-d69201ac3a2b-config\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.537985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.537678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2400c62-09d7-4e9a-9640-85ce68a09fb4-config\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.537985 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.537762 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:03.037743529 +0000 UTC m=+52.303839900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : configmap references non-existent config key: service-ca.crt Sep 10 02:34:02.537985 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.537786 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:34:02.537985 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.537849 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:03.03781811 +0000 UTC m=+52.303914493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : secret "router-metrics-certs-default" not found Sep 10 02:34:02.538551 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.538528 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2400c62-09d7-4e9a-9640-85ce68a09fb4-trusted-ca\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.539863 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.539472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" Sep 10 02:34:02.541462 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.541441 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-stats-auth\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.541551 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.541471 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-default-certificate\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.541885 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.541862 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2400c62-09d7-4e9a-9640-85ce68a09fb4-serving-cert\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.542714 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.542665 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6ea64165-5263-4e39-b47c-d69201ac3a2b-serving-cert\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.548472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.548419 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf7mw\" (UniqueName: \"kubernetes.io/projected/6ea64165-5263-4e39-b47c-d69201ac3a2b-kube-api-access-mf7mw\") pod \"service-ca-operator-859f94c4b-ngs6c\" (UID: \"6ea64165-5263-4e39-b47c-d69201ac3a2b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.548472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.548443 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-247v6\" (UniqueName: \"kubernetes.io/projected/a2400c62-09d7-4e9a-9640-85ce68a09fb4-kube-api-access-247v6\") pod \"console-operator-85b454874b-xr5ft\" (UID: \"a2400c62-09d7-4e9a-9640-85ce68a09fb4\") " pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.550924 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.550845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9z9\" (UniqueName: \"kubernetes.io/projected/d22aa7ab-c491-460d-8415-d452a6a64976-kube-api-access-cp9z9\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:02.593327 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.593288 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" Sep 10 02:34:02.609166 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.609140 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:02.683065 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.682525 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p"] Sep 10 02:34:02.686050 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:02.686026 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41df8304_9f36_45be_a5c7_696eb836d8e9.slice/crio-382fe89e18e534199ae10445a767b7d8d1e0b7d88db0b6bb6d6745e99e6272b8 WatchSource:0}: Error finding container 382fe89e18e534199ae10445a767b7d8d1e0b7d88db0b6bb6d6745e99e6272b8: Status 404 returned error can't find the container with id 382fe89e18e534199ae10445a767b7d8d1e0b7d88db0b6bb6d6745e99e6272b8 Sep 10 02:34:02.762430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.762400 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-djzb9"] Sep 10 02:34:02.765289 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:02.765264 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8516628f_ff2b_402d_985f_354faf0f1478.slice/crio-90c58efaf7b619ed3742de6f0b1f2fa6504dab1812f26be1319024de4fa6a5c9 WatchSource:0}: Error finding container 90c58efaf7b619ed3742de6f0b1f2fa6504dab1812f26be1319024de4fa6a5c9: Status 404 returned error can't find the container with id 90c58efaf7b619ed3742de6f0b1f2fa6504dab1812f26be1319024de4fa6a5c9 Sep 10 02:34:02.797138 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.797084 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg"] Sep 10 02:34:02.799015 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:02.798993 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a99e6f_fe48_48fc_a340_09ae9d1728fc.slice/crio-9967d2d50783388ba041a74d37ea361d3f0b8081fa1b2b9720a168682ad2a111 WatchSource:0}: Error finding container 9967d2d50783388ba041a74d37ea361d3f0b8081fa1b2b9720a168682ad2a111: Status 404 returned error can't find the container with id 9967d2d50783388ba041a74d37ea361d3f0b8081fa1b2b9720a168682ad2a111 Sep 10 02:34:02.824206 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.824163 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-bk7gd"] Sep 10 02:34:02.826370 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:02.826345 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92db0478_ff93_4fb6_a696_0885419be325.slice/crio-129deddcc88f8ba1b382cb896958fcb4bea03c0ba64b2b31523cd41432a8602e WatchSource:0}: Error finding container 129deddcc88f8ba1b382cb896958fcb4bea03c0ba64b2b31523cd41432a8602e: Status 404 returned error can't find the container with id 129deddcc88f8ba1b382cb896958fcb4bea03c0ba64b2b31523cd41432a8602e Sep 10 02:34:02.840558 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.840534 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:02.840690 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.840676 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:34:02.840741 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.840732 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls podName:301963d9-6567-47f5-9484-e9525db81b46 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:03.840713774 +0000 UTC m=+53.106810160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lnprl" (UID: "301963d9-6567-47f5-9484-e9525db81b46") : secret "samples-operator-tls" not found Sep 10 02:34:02.880129 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.880100 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c"] Sep 10 02:34:02.883444 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:02.883397 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea64165_5263_4e39_b47c_d69201ac3a2b.slice/crio-e08404d02687e59020618b7ad1adf7e06fd24ecfe4d9b7ed427c36fba16bf5e6 WatchSource:0}: Error finding container e08404d02687e59020618b7ad1adf7e06fd24ecfe4d9b7ed427c36fba16bf5e6: Status 404 returned error can't find the container with id e08404d02687e59020618b7ad1adf7e06fd24ecfe4d9b7ed427c36fba16bf5e6 Sep 10 02:34:02.900267 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.900243 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-xr5ft"] Sep 10 02:34:02.903027 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:02.903006 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2400c62_09d7_4e9a_9640_85ce68a09fb4.slice/crio-1385fcc9008439cc67c224183bf8c57597d9fff7bae1d20b290e11133ac91fea WatchSource:0}: Error finding container 1385fcc9008439cc67c224183bf8c57597d9fff7bae1d20b290e11133ac91fea: Status 404 returned error can't find the container with id 1385fcc9008439cc67c224183bf8c57597d9fff7bae1d20b290e11133ac91fea Sep 10 02:34:02.941023 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:02.940991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:02.941171 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.941148 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:02.941241 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:02.941215 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls podName:ffbf8b54-3714-486c-bc13-4eeb9cd28d26 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:03.941201114 +0000 UTC m=+53.207297483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-frrmp" (UID: "ffbf8b54-3714-486c-bc13-4eeb9cd28d26") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:03.042145 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.042108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:03.042145 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.042153 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:03.042346 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:03.042265 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:34:03.042346 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:03.042307 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:04.042289288 +0000 UTC m=+53.308385680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : configmap references non-existent config key: service-ca.crt Sep 10 02:34:03.042346 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:03.042324 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:04.04231608 +0000 UTC m=+53.308412449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : secret "router-metrics-certs-default" not found Sep 10 02:34:03.223376 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.223341 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:03.322570 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.322535 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:03.423297 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.423255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:03.504728 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.504640 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" event={"ID":"a2400c62-09d7-4e9a-9640-85ce68a09fb4","Type":"ContainerStarted","Data":"1385fcc9008439cc67c224183bf8c57597d9fff7bae1d20b290e11133ac91fea"} Sep 10 02:34:03.506795 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.506754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" event={"ID":"92db0478-ff93-4fb6-a696-0885419be325","Type":"ContainerStarted","Data":"129deddcc88f8ba1b382cb896958fcb4bea03c0ba64b2b31523cd41432a8602e"} Sep 10 02:34:03.508739 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.508706 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" event={"ID":"24a99e6f-fe48-48fc-a340-09ae9d1728fc","Type":"ContainerStarted","Data":"9967d2d50783388ba041a74d37ea361d3f0b8081fa1b2b9720a168682ad2a111"} Sep 10 02:34:03.510426 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.510403 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" event={"ID":"6ea64165-5263-4e39-b47c-d69201ac3a2b","Type":"ContainerStarted","Data":"e08404d02687e59020618b7ad1adf7e06fd24ecfe4d9b7ed427c36fba16bf5e6"} Sep 10 02:34:03.512376 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.512346 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" event={"ID":"8516628f-ff2b-402d-985f-354faf0f1478","Type":"ContainerStarted","Data":"90c58efaf7b619ed3742de6f0b1f2fa6504dab1812f26be1319024de4fa6a5c9"} Sep 10 02:34:03.513704 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.513649 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" event={"ID":"41df8304-9f36-45be-a5c7-696eb836d8e9","Type":"ContainerStarted","Data":"382fe89e18e534199ae10445a767b7d8d1e0b7d88db0b6bb6d6745e99e6272b8"} Sep 10 02:34:03.850429 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.850340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:03.850629 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:03.850612 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:34:03.850694 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:03.850681 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls podName:301963d9-6567-47f5-9484-e9525db81b46 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:05.850662713 +0000 UTC m=+55.116759105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lnprl" (UID: "301963d9-6567-47f5-9484-e9525db81b46") : secret "samples-operator-tls" not found Sep 10 02:34:03.951556 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:03.951514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:03.951766 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:03.951664 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:03.951766 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:03.951733 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls podName:ffbf8b54-3714-486c-bc13-4eeb9cd28d26 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:05.951714645 +0000 UTC m=+55.217811028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-frrmp" (UID: "ffbf8b54-3714-486c-bc13-4eeb9cd28d26") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:04.052709 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.052492 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:04.052709 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.052685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:04.052954 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:04.052808 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:34:04.052954 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:04.052886 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:06.052867256 +0000 UTC m=+55.318963639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : secret "router-metrics-certs-default" not found Sep 10 02:34:04.053592 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:04.053310 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:06.053291857 +0000 UTC m=+55.319388243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : configmap references non-existent config key: service-ca.crt Sep 10 02:34:04.247186 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.247049 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:04.345078 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.344986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:04.448731 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.448699 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:04.519692 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.519656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" event={"ID":"41df8304-9f36-45be-a5c7-696eb836d8e9","Type":"ContainerStarted","Data":"374a930fde14374fe2c8ae16b32b0a9c7777ddf055b8cbebaa825315234a5de6"} Sep 10 02:34:04.626438 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.626403 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:04.638933 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.638881 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-4c45p" podStartSLOduration=1.194121654 podStartE2EDuration="2.63885924s" totalImagesPullingTime="1.444737586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:04.538232868 +0000 UTC m=+53.804329260" watchObservedRunningTime="2026-09-10 02:34:04.63885924 +0000 UTC m=+53.904955632" Sep 10 02:34:04.640052 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.640030 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-99qz4"] Sep 10 02:34:04.643360 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.643340 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:04.646357 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.646329 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jml7x\"" Sep 10 02:34:04.646627 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.646607 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 02:34:04.646813 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.646796 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 02:34:04.656682 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.656658 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-99qz4"] Sep 10 02:34:04.727510 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.727374 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:04.760760 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.760700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9f5105d-d773-45f9-afc2-69a1f0fbc218-nginx-conf\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:04.760939 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.760763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:04.862234 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.861807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:04.862234 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:04.861970 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 02:34:04.862234 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:04.862049 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert podName:a9f5105d-d773-45f9-afc2-69a1f0fbc218 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:05.362025509 +0000 UTC m=+54.628121890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-99qz4" (UID: "a9f5105d-d773-45f9-afc2-69a1f0fbc218") : secret "networking-console-plugin-cert" not found Sep 10 02:34:04.862234 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.862170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9f5105d-d773-45f9-afc2-69a1f0fbc218-nginx-conf\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:04.863058 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:04.862893 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9f5105d-d773-45f9-afc2-69a1f0fbc218-nginx-conf\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:05.200439 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:05.200404 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:05.366549 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:05.366509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:05.366723 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:05.366679 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 02:34:05.366770 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:05.366746 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert podName:a9f5105d-d773-45f9-afc2-69a1f0fbc218 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:06.366729153 +0000 UTC m=+55.632825536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-99qz4" (UID: "a9f5105d-d773-45f9-afc2-69a1f0fbc218") : secret "networking-console-plugin-cert" not found Sep 10 02:34:05.589297 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:05.589216 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:05.646372 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:05.646340 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:05.870064 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:05.869979 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:05.870231 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:05.870140 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:34:05.870231 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:05.870221 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls podName:301963d9-6567-47f5-9484-e9525db81b46 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:09.870202417 +0000 UTC m=+59.136298803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lnprl" (UID: "301963d9-6567-47f5-9484-e9525db81b46") : secret "samples-operator-tls" not found Sep 10 02:34:05.971274 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:05.971233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:05.971427 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:05.971381 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:05.971478 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:05.971451 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls podName:ffbf8b54-3714-486c-bc13-4eeb9cd28d26 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:09.971432964 +0000 UTC m=+59.237529353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-frrmp" (UID: "ffbf8b54-3714-486c-bc13-4eeb9cd28d26") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:06.072038 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:06.072000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:06.072212 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:06.072048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:06.072212 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:06.072162 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:10.072149199 +0000 UTC m=+59.338245568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : configmap references non-existent config key: service-ca.crt Sep 10 02:34:06.072212 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:06.072194 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:34:06.072384 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:06.072265 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:10.072247786 +0000 UTC m=+59.338344170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : secret "router-metrics-certs-default" not found Sep 10 02:34:06.375875 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:06.375811 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:06.376057 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:06.376001 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 02:34:06.376115 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:06.376077 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert podName:a9f5105d-d773-45f9-afc2-69a1f0fbc218 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:08.3760579 +0000 UTC m=+57.642154285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-99qz4" (UID: "a9f5105d-d773-45f9-afc2-69a1f0fbc218") : secret "networking-console-plugin-cert" not found Sep 10 02:34:08.394676 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.394631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:08.395179 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:08.394780 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 02:34:08.395179 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:08.394868 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert podName:a9f5105d-d773-45f9-afc2-69a1f0fbc218 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:12.394848105 +0000 UTC m=+61.660944491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-99qz4" (UID: "a9f5105d-d773-45f9-afc2-69a1f0fbc218") : secret "networking-console-plugin-cert" not found Sep 10 02:34:08.533784 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.533739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" event={"ID":"8516628f-ff2b-402d-985f-354faf0f1478","Type":"ContainerStarted","Data":"d425798928644a36655a200a1607bdfca31c1c00d67ddd401747e3370ad583c3"} Sep 10 02:34:08.536375 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.536347 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/0.log" Sep 10 02:34:08.536504 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.536383 2570 generic.go:356] "Generic (PLEG): container finished" podID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" containerID="b31993e3837396b0316a14999d7618fb5b82018fa9472ff3e1b46e1ba5974d0a" exitCode=255 Sep 10 02:34:08.536596 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.536573 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" event={"ID":"a2400c62-09d7-4e9a-9640-85ce68a09fb4","Type":"ContainerDied","Data":"b31993e3837396b0316a14999d7618fb5b82018fa9472ff3e1b46e1ba5974d0a"} Sep 10 02:34:08.536719 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.536702 2570 scope.go:117] "RemoveContainer" containerID="b31993e3837396b0316a14999d7618fb5b82018fa9472ff3e1b46e1ba5974d0a" Sep 10 02:34:08.538495 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.538474 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" event={"ID":"92db0478-ff93-4fb6-a696-0885419be325","Type":"ContainerStarted","Data":"dad8dd141e6f4e95a591804f52245efc7ed33bdc4e3793a7f6c6b8cf82927a8d"} Sep 10 02:34:08.540634 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.540595 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" event={"ID":"24a99e6f-fe48-48fc-a340-09ae9d1728fc","Type":"ContainerStarted","Data":"f2b162a1188606425646050887eea05980aee3b61ff3d4aed514928861d8c32f"} Sep 10 02:34:08.542201 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.542169 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" event={"ID":"6ea64165-5263-4e39-b47c-d69201ac3a2b","Type":"ContainerStarted","Data":"e0a9e5e5b432397f3bf05d14aa21b3912c36c9572a8a966412d46961c8a0b2f5"} Sep 10 02:34:08.550343 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.550303 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-djzb9" podStartSLOduration=1.36037369 podStartE2EDuration="6.550287367s" totalImagesPullingTime="5.189913677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:08.549789576 +0000 UTC m=+57.815885968" watchObservedRunningTime="2026-09-10 02:34:08.550287367 +0000 UTC m=+57.816383762" Sep 10 02:34:08.568548 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.568506 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" podStartSLOduration=1.423092093 podStartE2EDuration="6.56849472s" totalImagesPullingTime="5.145402627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:08.567191299 +0000 UTC m=+57.833287692" watchObservedRunningTime="2026-09-10 02:34:08.56849472 +0000 UTC m=+57.834591110" Sep 10 02:34:08.599498 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.599437 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" podStartSLOduration=1.5358744720000002 podStartE2EDuration="6.599420207s" totalImagesPullingTime="5.063545735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:08.598781733 +0000 UTC m=+57.864878138" watchObservedRunningTime="2026-09-10 02:34:08.599420207 +0000 UTC m=+57.865516599" Sep 10 02:34:08.617385 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.617342 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" podStartSLOduration=1.500101771 podStartE2EDuration="6.61732612s" totalImagesPullingTime="5.117224349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:08.616174383 +0000 UTC m=+57.882270775" watchObservedRunningTime="2026-09-10 02:34:08.61732612 +0000 UTC m=+57.883422542" Sep 10 02:34:08.763344 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.763306 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:08.793595 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.793562 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx"] Sep 10 02:34:08.797054 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.797034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" Sep 10 02:34:08.800326 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.800304 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b947b\"" Sep 10 02:34:08.800466 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.800304 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 02:34:08.801627 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.801610 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 02:34:08.813126 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.813104 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx"] Sep 10 02:34:08.856108 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.856060 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:08.899167 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.899143 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-252f6\" (UniqueName: \"kubernetes.io/projected/efa04a19-d382-4e5e-9cdb-a48b40b57220-kube-api-access-252f6\") pod \"migrator-fdbb8fdff-zknhx\" (UID: \"efa04a19-d382-4e5e-9cdb-a48b40b57220\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" Sep 10 02:34:08.956424 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:08.956400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:09.000435 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.000410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-252f6\" (UniqueName: \"kubernetes.io/projected/efa04a19-d382-4e5e-9cdb-a48b40b57220-kube-api-access-252f6\") pod \"migrator-fdbb8fdff-zknhx\" (UID: \"efa04a19-d382-4e5e-9cdb-a48b40b57220\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" Sep 10 02:34:09.009303 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.009280 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-252f6\" (UniqueName: \"kubernetes.io/projected/efa04a19-d382-4e5e-9cdb-a48b40b57220-kube-api-access-252f6\") pod \"migrator-fdbb8fdff-zknhx\" (UID: \"efa04a19-d382-4e5e-9cdb-a48b40b57220\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" Sep 10 02:34:09.106186 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.106147 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" Sep 10 02:34:09.404756 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.404609 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx"] Sep 10 02:34:09.407575 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:09.407539 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa04a19_d382_4e5e_9cdb_a48b40b57220.slice/crio-46c2a794582c828f5361353fdb5eea58440b7b73b1aed70dd5e9975ad85873f0 WatchSource:0}: Error finding container 46c2a794582c828f5361353fdb5eea58440b7b73b1aed70dd5e9975ad85873f0: Status 404 returned error can't find the container with id 46c2a794582c828f5361353fdb5eea58440b7b73b1aed70dd5e9975ad85873f0 Sep 10 02:34:09.499874 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.499816 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8ssl7"] Sep 10 02:34:09.526957 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.526922 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:09.529143 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.529117 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8ssl7"] Sep 10 02:34:09.529272 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.529256 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.531840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.531798 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 02:34:09.531959 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.531845 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9mpm\"" Sep 10 02:34:09.531959 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.531806 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 02:34:09.549807 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.549782 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/1.log" Sep 10 02:34:09.550255 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.550237 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/0.log" Sep 10 02:34:09.550355 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.550273 2570 generic.go:356] "Generic (PLEG): container finished" podID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" containerID="802ad723ac0f79c09c65364f185f8a7d9262bafeb2fe5d47dd36bf505b9dc258" exitCode=255 Sep 10 02:34:09.550419 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.550368 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" event={"ID":"a2400c62-09d7-4e9a-9640-85ce68a09fb4","Type":"ContainerDied","Data":"802ad723ac0f79c09c65364f185f8a7d9262bafeb2fe5d47dd36bf505b9dc258"} Sep 10 02:34:09.550511 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.550456 2570 scope.go:117] "RemoveContainer" containerID="b31993e3837396b0316a14999d7618fb5b82018fa9472ff3e1b46e1ba5974d0a" Sep 10 02:34:09.550569 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.550538 2570 scope.go:117] "RemoveContainer" containerID="802ad723ac0f79c09c65364f185f8a7d9262bafeb2fe5d47dd36bf505b9dc258" Sep 10 02:34:09.550954 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:09.550718 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-xr5ft_openshift-console-operator(a2400c62-09d7-4e9a-9640-85ce68a09fb4)\"" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" podUID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" Sep 10 02:34:09.551733 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.551577 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" event={"ID":"efa04a19-d382-4e5e-9cdb-a48b40b57220","Type":"ContainerStarted","Data":"46c2a794582c828f5361353fdb5eea58440b7b73b1aed70dd5e9975ad85873f0"} Sep 10 02:34:09.588720 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.588698 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:09.608982 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.608955 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.609173 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.609149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsl5k\" (UniqueName: \"kubernetes.io/projected/6e07c18b-4023-461c-890c-dd853cda3975-kube-api-access-hsl5k\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.609263 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.609246 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e07c18b-4023-461c-890c-dd853cda3975-crio-socket\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.609899 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.609463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e07c18b-4023-461c-890c-dd853cda3975-data-volume\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.609899 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.609630 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.710556 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.710528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e07c18b-4023-461c-890c-dd853cda3975-data-volume\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.710684 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.710673 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.710737 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.710714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.710783 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.710742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsl5k\" (UniqueName: \"kubernetes.io/projected/6e07c18b-4023-461c-890c-dd853cda3975-kube-api-access-hsl5k\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.710871 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:09.710851 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:34:09.710916 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.710868 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e07c18b-4023-461c-890c-dd853cda3975-data-volume\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.710916 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.710877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e07c18b-4023-461c-890c-dd853cda3975-crio-socket\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.711005 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:09.710921 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:10.210901519 +0000 UTC m=+59.476997903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : secret "insights-runtime-extractor-tls" not found Sep 10 02:34:09.711005 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.710952 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6e07c18b-4023-461c-890c-dd853cda3975-crio-socket\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.711005 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:09.710958 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:34:09.711005 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:09.711003 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:10.210995614 +0000 UTC m=+59.477091983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : configmap "kube-rbac-proxy" not found Sep 10 02:34:09.719406 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.719389 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsl5k\" (UniqueName: \"kubernetes.io/projected/6e07c18b-4023-461c-890c-dd853cda3975-kube-api-access-hsl5k\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:09.912793 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:09.912757 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:09.912995 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:09.912928 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 02:34:09.913060 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:09.913002 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls podName:301963d9-6567-47f5-9484-e9525db81b46 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:17.912982227 +0000 UTC m=+67.179078596 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-lnprl" (UID: "301963d9-6567-47f5-9484-e9525db81b46") : secret "samples-operator-tls" not found Sep 10 02:34:10.014004 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.013895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:10.014163 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.014083 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:10.014163 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.014156 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls podName:ffbf8b54-3714-486c-bc13-4eeb9cd28d26 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:18.014137486 +0000 UTC m=+67.280233872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-frrmp" (UID: "ffbf8b54-3714-486c-bc13-4eeb9cd28d26") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:10.014842 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.014795 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:10.114611 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.114579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:10.114815 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.114724 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:18.114704009 +0000 UTC m=+67.380800393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : configmap references non-existent config key: service-ca.crt Sep 10 02:34:10.114982 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.114867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:10.115053 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.114984 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:34:10.115053 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.115023 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:18.115011102 +0000 UTC m=+67.381107485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : secret "router-metrics-certs-default" not found Sep 10 02:34:10.115530 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.115503 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:10.215538 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.215499 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:10.215677 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.215640 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:34:10.215735 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.215691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:10.215735 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.215717 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:11.215697917 +0000 UTC m=+60.481794309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : configmap "kube-rbac-proxy" not found Sep 10 02:34:10.215851 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.215784 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:34:10.215891 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.215865 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:11.215848733 +0000 UTC m=+60.481945115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : secret "insights-runtime-extractor-tls" not found Sep 10 02:34:10.294082 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.293999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:10.394671 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.394640 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:10.557419 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.557358 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/1.log" Sep 10 02:34:10.557789 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:10.557697 2570 scope.go:117] "RemoveContainer" containerID="802ad723ac0f79c09c65364f185f8a7d9262bafeb2fe5d47dd36bf505b9dc258" Sep 10 02:34:10.557911 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:10.557890 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-xr5ft_openshift-console-operator(a2400c62-09d7-4e9a-9640-85ce68a09fb4)\"" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" podUID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" Sep 10 02:34:11.224980 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.224952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:11.225155 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.225022 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:11.225155 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:11.225064 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:34:11.225155 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:11.225130 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:13.225108746 +0000 UTC m=+62.491205115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : configmap "kube-rbac-proxy" not found Sep 10 02:34:11.225323 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:11.225186 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:34:11.225323 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:11.225223 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:13.225211221 +0000 UTC m=+62.491307604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : secret "insights-runtime-extractor-tls" not found Sep 10 02:34:11.443719 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.443692 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs6vw" Sep 10 02:34:11.481114 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.481052 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvvrb_55541690-c514-404a-8037-7bb94a66eaf5/dns-node-resolver/0.log" Sep 10 02:34:11.562293 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.562267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" event={"ID":"efa04a19-d382-4e5e-9cdb-a48b40b57220","Type":"ContainerStarted","Data":"12d930a3af790b3aede369346dd266c0a4d7cb0e39d2a4517c60cb608fc6894d"} Sep 10 02:34:11.562625 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.562312 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" event={"ID":"efa04a19-d382-4e5e-9cdb-a48b40b57220","Type":"ContainerStarted","Data":"cef81b87dcf49e6192403a58a52047fa504c83c2a26c5da21f33899a3a961838"} Sep 10 02:34:11.580265 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.580220 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-zknhx" podStartSLOduration=2.3032202440000002 podStartE2EDuration="3.580203899s" totalImagesPullingTime="1.276983655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:11.579141374 +0000 UTC m=+60.845237804" watchObservedRunningTime="2026-09-10 02:34:11.580203899 +0000 UTC m=+60.846300289" Sep 10 02:34:11.630465 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.630442 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:11.729649 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.729620 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:11.886393 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.886320 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-s6v7q"] Sep 10 02:34:11.890453 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.890434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:11.892772 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.892748 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 02:34:11.892928 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.892905 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 02:34:11.892928 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.892905 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 02:34:11.893086 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.892964 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 02:34:11.893086 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.893028 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7s556\"" Sep 10 02:34:11.899644 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.899624 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-s6v7q"] Sep 10 02:34:11.923876 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.923849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:11.929871 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.929851 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4449eecb-56ae-4e5d-9010-1489827a597e-signing-cabundle\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:11.929985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.929969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wzz\" (UniqueName: \"kubernetes.io/projected/4449eecb-56ae-4e5d-9010-1489827a597e-kube-api-access-t7wzz\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:11.930033 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:11.930021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4449eecb-56ae-4e5d-9010-1489827a597e-signing-key\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.025232 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.025195 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:12.031217 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.031196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wzz\" (UniqueName: \"kubernetes.io/projected/4449eecb-56ae-4e5d-9010-1489827a597e-kube-api-access-t7wzz\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.031295 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.031253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4449eecb-56ae-4e5d-9010-1489827a597e-signing-key\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.031355 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.031342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4449eecb-56ae-4e5d-9010-1489827a597e-signing-cabundle\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.031964 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.031942 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4449eecb-56ae-4e5d-9010-1489827a597e-signing-cabundle\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.033481 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.033463 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4449eecb-56ae-4e5d-9010-1489827a597e-signing-key\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.043331 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.043298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wzz\" (UniqueName: \"kubernetes.io/projected/4449eecb-56ae-4e5d-9010-1489827a597e-kube-api-access-t7wzz\") pod \"service-ca-75d99c484-s6v7q\" (UID: \"4449eecb-56ae-4e5d-9010-1489827a597e\") " pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.199075 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.199044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-s6v7q" Sep 10 02:34:12.434784 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.434752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:12.434984 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:12.434946 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 02:34:12.435048 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:12.435020 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert podName:a9f5105d-d773-45f9-afc2-69a1f0fbc218 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:20.434999993 +0000 UTC m=+69.701096381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-99qz4" (UID: "a9f5105d-d773-45f9-afc2-69a1f0fbc218") : secret "networking-console-plugin-cert" not found Sep 10 02:34:12.468942 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.468814 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-s6v7q"] Sep 10 02:34:12.471469 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:12.471446 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4449eecb_56ae_4e5d_9010_1489827a597e.slice/crio-a17ded7e40d0d47f174d8cc961a77780e3bd40234d76eafa730ec3f2d7b5dc3b WatchSource:0}: Error finding container a17ded7e40d0d47f174d8cc961a77780e3bd40234d76eafa730ec3f2d7b5dc3b: Status 404 returned error can't find the container with id a17ded7e40d0d47f174d8cc961a77780e3bd40234d76eafa730ec3f2d7b5dc3b Sep 10 02:34:12.572615 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.572585 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-s6v7q" event={"ID":"4449eecb-56ae-4e5d-9010-1489827a597e","Type":"ContainerStarted","Data":"d74a960b3653dc3aefe7d6340d22ca29462e0330dee69f2c0d746735f87b6574"} Sep 10 02:34:12.572986 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.572666 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-s6v7q" event={"ID":"4449eecb-56ae-4e5d-9010-1489827a597e","Type":"ContainerStarted","Data":"a17ded7e40d0d47f174d8cc961a77780e3bd40234d76eafa730ec3f2d7b5dc3b"} Sep 10 02:34:12.592273 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.592226 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-s6v7q" podStartSLOduration=1.592212492 podStartE2EDuration="1.592212492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:12.590985664 +0000 UTC m=+61.857082079" watchObservedRunningTime="2026-09-10 02:34:12.592212492 +0000 UTC m=+61.858308882" Sep 10 02:34:12.609420 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.609400 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:12.609500 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.609437 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:12.609779 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.609762 2570 scope.go:117] "RemoveContainer" containerID="802ad723ac0f79c09c65364f185f8a7d9262bafeb2fe5d47dd36bf505b9dc258" Sep 10 02:34:12.610005 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:12.609983 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-xr5ft_openshift-console-operator(a2400c62-09d7-4e9a-9640-85ce68a09fb4)\"" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" podUID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" Sep 10 02:34:12.682074 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.682049 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w7fk8_1c19c39b-51ec-4ed1-8be8-f99fc93448a5/node-ca/0.log" Sep 10 02:34:12.899731 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.899658 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:13.000018 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:12.999980 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:13.083638 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.083613 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-whf7d_9d0bb233-da91-40b6-963b-80352b7f96ee/serve-healthcheck-canary/0.log" Sep 10 02:34:13.127486 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.127446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:13.200091 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.200058 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:13.242743 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.242711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:13.242917 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.242853 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:13.242917 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:13.242886 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:34:13.243024 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:13.242934 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:34:13.243024 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:13.242962 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:17.242940363 +0000 UTC m=+66.509036751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : secret "insights-runtime-extractor-tls" not found Sep 10 02:34:13.243024 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:13.242984 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:17.242973905 +0000 UTC m=+66.509070280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : configmap "kube-rbac-proxy" not found Sep 10 02:34:13.281802 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.281775 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-zknhx_efa04a19-d382-4e5e-9cdb-a48b40b57220/migrator/0.log" Sep 10 02:34:13.300735 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.300690 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:13.482510 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.482432 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-zknhx_efa04a19-d382-4e5e-9cdb-a48b40b57220/graceful-termination/0.log" Sep 10 02:34:13.683283 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:13.683250 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-szhxg_24a99e6f-fe48-48fc-a340-09ae9d1728fc/kube-storage-version-migrator-operator/0.log" Sep 10 02:34:14.207232 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:14.207198 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:14.307972 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:14.307933 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:14.408625 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:14.408593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:14.509132 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:14.509055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:14.611098 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:14.611057 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:15.095912 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:15.095879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:15.708410 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:15.708379 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:15.808748 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:15.808712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:16.109224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:16.109128 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:16.509586 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:16.509549 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:16.903551 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:16.903470 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:16.976784 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:16.976754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:34:16.980215 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:16.980197 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 02:34:16.987262 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:16.987246 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 02:34:16.987333 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:16.987302 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs podName:253d3186-27d1-4b7e-b48a-303871be8896 nodeName:}" failed. No retries permitted until 2026-09-10 02:35:20.987287855 +0000 UTC m=+130.253384224 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs") pod "network-metrics-daemon-6hhxs" (UID: "253d3186-27d1-4b7e-b48a-303871be8896") : secret "metrics-daemon-secret" not found Sep 10 02:34:17.003871 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.003801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:17.078078 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.078044 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:34:17.078078 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.078083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:34:17.080504 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.080486 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd6b\" (UniqueName: \"kubernetes.io/projected/6c26c64a-148b-430a-b8a3-26ab98b62904-kube-api-access-gvd6b\") pod \"network-check-target-9rzqt\" (UID: \"6c26c64a-148b-430a-b8a3-26ab98b62904\") " pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:34:17.080736 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.080721 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 02:34:17.090912 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.090892 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3902b22c-0dda-4604-a7d8-439602edd83e-original-pull-secret\") pod \"global-pull-secret-syncer-sg8dz\" (UID: \"3902b22c-0dda-4604-a7d8-439602edd83e\") " pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:34:17.110223 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.110194 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:17.134103 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.134085 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sg8dz" Sep 10 02:34:17.251549 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.251524 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tcpg8\"" Sep 10 02:34:17.259917 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.259867 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:34:17.280409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.280384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:17.280498 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.280438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:17.280540 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.280502 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 02:34:17.280577 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.280556 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:25.280541955 +0000 UTC m=+74.546638466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : configmap "kube-rbac-proxy" not found Sep 10 02:34:17.280651 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.280631 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 02:34:17.280713 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.280702 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls podName:6e07c18b-4023-461c-890c-dd853cda3975 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:25.280683717 +0000 UTC m=+74.546780098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ssl7" (UID: "6e07c18b-4023-461c-890c-dd853cda3975") : secret "insights-runtime-extractor-tls" not found Sep 10 02:34:17.430453 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.430409 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sg8dz"] Sep 10 02:34:17.434132 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:17.434106 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3902b22c_0dda_4604_a7d8_439602edd83e.slice/crio-0842b6c15d677433fa9772315a293249111122892849b7327a58f5c6b2a5150f WatchSource:0}: Error finding container 0842b6c15d677433fa9772315a293249111122892849b7327a58f5c6b2a5150f: Status 404 returned error can't find the container with id 0842b6c15d677433fa9772315a293249111122892849b7327a58f5c6b2a5150f Sep 10 02:34:17.481632 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.481603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:34:17.481726 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.481662 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:34:17.481793 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.481763 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 02:34:17.481793 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.481779 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b47995f-scq9k: secret "image-registry-tls" not found Sep 10 02:34:17.481912 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.481847 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls podName:dc7a6fc4-786a-4e2e-84d9-283835e132ed nodeName:}" failed. No retries permitted until 2026-09-10 02:34:49.481816731 +0000 UTC m=+98.747913113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls") pod "image-registry-5b47995f-scq9k" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed") : secret "image-registry-tls" not found Sep 10 02:34:17.481912 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.481858 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 02:34:17.482005 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:17.481923 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls podName:d322117c-a2b5-4598-8c62-31ab1911f3cd nodeName:}" failed. No retries permitted until 2026-09-10 02:34:49.481902831 +0000 UTC m=+98.747999217 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls") pod "dns-default-v8xtg" (UID: "d322117c-a2b5-4598-8c62-31ab1911f3cd") : secret "dns-default-metrics-tls" not found Sep 10 02:34:17.504418 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.504386 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:17.550315 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.550288 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9rzqt"] Sep 10 02:34:17.562737 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:17.562713 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c26c64a_148b_430a_b8a3_26ab98b62904.slice/crio-c5c1cfea278f5ba0b60e66986fd30b5a30d75fd6d7cec0c9892d6bf5f0ce5ce2 WatchSource:0}: Error finding container c5c1cfea278f5ba0b60e66986fd30b5a30d75fd6d7cec0c9892d6bf5f0ce5ce2: Status 404 returned error can't find the container with id c5c1cfea278f5ba0b60e66986fd30b5a30d75fd6d7cec0c9892d6bf5f0ce5ce2 Sep 10 02:34:17.591108 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.591075 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9rzqt" event={"ID":"6c26c64a-148b-430a-b8a3-26ab98b62904","Type":"ContainerStarted","Data":"c5c1cfea278f5ba0b60e66986fd30b5a30d75fd6d7cec0c9892d6bf5f0ce5ce2"} Sep 10 02:34:17.592218 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.592196 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sg8dz" event={"ID":"3902b22c-0dda-4604-a7d8-439602edd83e","Type":"ContainerStarted","Data":"0842b6c15d677433fa9772315a293249111122892849b7327a58f5c6b2a5150f"} Sep 10 02:34:17.902525 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.902481 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:17.987384 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.987305 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:17.989631 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:17.989607 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/301963d9-6567-47f5-9484-e9525db81b46-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-lnprl\" (UID: \"301963d9-6567-47f5-9484-e9525db81b46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:18.025698 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.025674 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mmd8f\"" Sep 10 02:34:18.033717 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.033702 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" Sep 10 02:34:18.089164 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.088926 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:18.089286 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:18.089179 2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:18.089286 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:18.089252 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls podName:ffbf8b54-3714-486c-bc13-4eeb9cd28d26 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:34.089232274 +0000 UTC m=+83.355328665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-frrmp" (UID: "ffbf8b54-3714-486c-bc13-4eeb9cd28d26") : secret "cluster-monitoring-operator-tls" not found Sep 10 02:34:18.099493 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.099464 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:18.189974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.189933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:18.190430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.189987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:18.190430 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:18.190118 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:34.190096905 +0000 UTC m=+83.456193280 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : configmap references non-existent config key: service-ca.crt Sep 10 02:34:18.190430 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:18.190154 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 02:34:18.190430 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:18.190211 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs podName:d22aa7ab-c491-460d-8415-d452a6a64976 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:34.190195736 +0000 UTC m=+83.456292122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs") pod "router-default-6fb5dd5bbd-khhbd" (UID: "d22aa7ab-c491-460d-8415-d452a6a64976") : secret "router-metrics-certs-default" not found Sep 10 02:34:18.195596 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.195566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:18.302397 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.302326 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl"] Sep 10 02:34:18.499883 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.499844 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:18.598190 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.598111 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9rzqt" event={"ID":"6c26c64a-148b-430a-b8a3-26ab98b62904","Type":"ContainerStarted","Data":"759029650eecd12c95933daea9899c67dcda09709ac5a2a1c1b06ef4e8dc1e9d"} Sep 10 02:34:18.598456 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.598239 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:34:18.599627 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.599599 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" event={"ID":"301963d9-6567-47f5-9484-e9525db81b46","Type":"ContainerStarted","Data":"c5f4bba93d0fe0972d5540d74669f2ef050fbb6f3da9142282a5d8e85382cb58"} Sep 10 02:34:18.616866 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.616809 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9rzqt" podStartSLOduration=67.616791613 podStartE2EDuration="1m7.616791613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:18.614744301 +0000 UTC m=+67.880840705" watchObservedRunningTime="2026-09-10 02:34:18.616791613 +0000 UTC m=+67.882888008" Sep 10 02:34:18.895670 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.895581 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:18.967985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:18.967948 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:19.101451 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:19.101390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:19.470849 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:19.467268 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:19.698090 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:19.698051 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:19.898421 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:19.898337 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:20.304848 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:20.304683 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:20.404464 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:20.404427 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:20.504432 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:20.504400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:20.511629 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:20.511600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:20.511774 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:20.511753 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 02:34:20.511843 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:20.511805 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert podName:a9f5105d-d773-45f9-afc2-69a1f0fbc218 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:36.511788567 +0000 UTC m=+85.777884937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-99qz4" (UID: "a9f5105d-d773-45f9-afc2-69a1f0fbc218") : secret "networking-console-plugin-cert" not found Sep 10 02:34:20.604791 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:20.604693 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:20.897876 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:20.897781 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:21.095345 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:21.095311 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:21.493925 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:21.493886 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:21.903270 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:21.903181 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:22.100197 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.100151 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:22.496954 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.496920 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:22.615957 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.615920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" event={"ID":"301963d9-6567-47f5-9484-e9525db81b46","Type":"ContainerStarted","Data":"4bac2ae9c5c5a112d4c26f7aa00424f821c44da3b644dbb064e8a41c91b3a426"} Sep 10 02:34:22.616115 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.616005 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" event={"ID":"301963d9-6567-47f5-9484-e9525db81b46","Type":"ContainerStarted","Data":"713853c2d5bb587eed03a53578a78d5bd410826d4b92869340464ac99323bf67"} Sep 10 02:34:22.617359 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.617333 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sg8dz" event={"ID":"3902b22c-0dda-4604-a7d8-439602edd83e","Type":"ContainerStarted","Data":"c3a027ff46ba6698ef0b2141366b3652b8a6a76a6255154fe712cb04be91a7b3"} Sep 10 02:34:22.631940 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.631896 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-lnprl" podStartSLOduration=17.065674083 podStartE2EDuration="20.631881411s" totalImagesPullingTime="3.566207328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:22.630946632 +0000 UTC m=+71.897043025" watchObservedRunningTime="2026-09-10 02:34:22.631881411 +0000 UTC m=+71.897977803" Sep 10 02:34:22.645358 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.645315 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sg8dz" podStartSLOduration=67.15490719 podStartE2EDuration="1m11.645301615s" totalImagesPullingTime="4.490394425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:22.645124896 +0000 UTC m=+71.911221290" watchObservedRunningTime="2026-09-10 02:34:22.645301615 +0000 UTC m=+71.911398006" Sep 10 02:34:22.898863 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:22.898758 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:23.297453 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:23.297415 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:23.699994 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:23.699957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:23.901841 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:23.901801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:24.222001 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.221978 2570 scope.go:117] "RemoveContainer" containerID="802ad723ac0f79c09c65364f185f8a7d9262bafeb2fe5d47dd36bf505b9dc258" Sep 10 02:34:24.498276 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.498165 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:24.626293 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.626269 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:34:24.626679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.626663 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/1.log" Sep 10 02:34:24.626726 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.626700 2570 generic.go:356] "Generic (PLEG): container finished" podID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" containerID="668d9682183bf853bf06c5cd9e974c5d3db4c1d928f48865e633858eef31f2e3" exitCode=255 Sep 10 02:34:24.626759 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.626728 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" event={"ID":"a2400c62-09d7-4e9a-9640-85ce68a09fb4","Type":"ContainerDied","Data":"668d9682183bf853bf06c5cd9e974c5d3db4c1d928f48865e633858eef31f2e3"} Sep 10 02:34:24.626797 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.626768 2570 scope.go:117] "RemoveContainer" containerID="802ad723ac0f79c09c65364f185f8a7d9262bafeb2fe5d47dd36bf505b9dc258" Sep 10 02:34:24.627098 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.627075 2570 scope.go:117] "RemoveContainer" containerID="668d9682183bf853bf06c5cd9e974c5d3db4c1d928f48865e633858eef31f2e3" Sep 10 02:34:24.627253 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:24.627235 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-xr5ft_openshift-console-operator(a2400c62-09d7-4e9a-9640-85ce68a09fb4)\"" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" podUID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" Sep 10 02:34:24.697631 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.697587 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:24.896789 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:24.896699 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:25.354086 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.354046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:25.354286 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.354122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:25.354629 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.354608 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6e07c18b-4023-461c-890c-dd853cda3975-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:25.356476 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.356446 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6e07c18b-4023-461c-890c-dd853cda3975-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ssl7\" (UID: \"6e07c18b-4023-461c-890c-dd853cda3975\") " pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:25.451453 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.451422 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9mpm\"" Sep 10 02:34:25.459372 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.459347 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8ssl7" Sep 10 02:34:25.631794 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.631715 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:34:25.728670 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:25.728635 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8ssl7"] Sep 10 02:34:25.731488 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:25.731463 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e07c18b_4023_461c_890c_dd853cda3975.slice/crio-2b79c161bedf21ea47625c8bfd2efb779bd94e99c3e8d1c209cbf45a8c8e3aa7 WatchSource:0}: Error finding container 2b79c161bedf21ea47625c8bfd2efb779bd94e99c3e8d1c209cbf45a8c8e3aa7: Status 404 returned error can't find the container with id 2b79c161bedf21ea47625c8bfd2efb779bd94e99c3e8d1c209cbf45a8c8e3aa7 Sep 10 02:34:26.638188 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:26.638164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ssl7" event={"ID":"6e07c18b-4023-461c-890c-dd853cda3975","Type":"ContainerStarted","Data":"278de33704e87ca144752e5456f52710f0fdbd4f578508ec902e5eaa8eaea24a"} Sep 10 02:34:26.638486 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:26.638213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ssl7" event={"ID":"6e07c18b-4023-461c-890c-dd853cda3975","Type":"ContainerStarted","Data":"2b79c161bedf21ea47625c8bfd2efb779bd94e99c3e8d1c209cbf45a8c8e3aa7"} Sep 10 02:34:27.644680 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:27.644644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ssl7" event={"ID":"6e07c18b-4023-461c-890c-dd853cda3975","Type":"ContainerStarted","Data":"e76942031508ab0060d63ce21b8478820986795fa4b1026ffe163b99e129b863"} Sep 10 02:34:28.650224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:28.650138 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ssl7" event={"ID":"6e07c18b-4023-461c-890c-dd853cda3975","Type":"ContainerStarted","Data":"000e9c775a6cd029aea5846ff1958743d23086100b3fc71355628a635e4cb6e9"} Sep 10 02:34:28.669113 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:28.669059 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8ssl7" podStartSLOduration=17.326106835 podStartE2EDuration="19.66904364s" totalImagesPullingTime="2.342936805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:34:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:28.667404303 +0000 UTC m=+77.933500696" watchObservedRunningTime="2026-09-10 02:34:28.66904364 +0000 UTC m=+77.935140031" Sep 10 02:34:29.790401 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:29.790365 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:29.847219 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:29.847188 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:30.859618 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.859585 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d"] Sep 10 02:34:30.862752 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.862737 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:30.866580 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.866558 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 02:34:30.866580 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.866571 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 02:34:30.866750 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.866571 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 02:34:30.866750 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.866619 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 10 02:34:30.868843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.868794 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht"] Sep 10 02:34:30.872560 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.872543 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb"] Sep 10 02:34:30.872666 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.872650 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:30.876405 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.876378 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 10 02:34:30.876636 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.876619 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-tzkm8\"" Sep 10 02:34:30.878801 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.878771 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d"] Sep 10 02:34:30.878940 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.878919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:30.881903 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.881879 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 02:34:30.882288 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.882231 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 02:34:30.882545 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.882404 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 02:34:30.882887 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.882851 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 02:34:30.884868 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.884456 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht"] Sep 10 02:34:30.889814 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.889791 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb"] Sep 10 02:34:30.975104 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:30.975072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:31.001777 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.001745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f91b21d-a7d3-4670-8701-8d694f84394a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht\" (UID: \"8f91b21d-a7d3-4670-8701-8d694f84394a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:31.001777 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.001776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.001990 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.001795 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clfv\" (UniqueName: \"kubernetes.io/projected/8f91b21d-a7d3-4670-8701-8d694f84394a-kube-api-access-4clfv\") pod \"managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht\" (UID: \"8f91b21d-a7d3-4670-8701-8d694f84394a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:31.001990 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.001819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4tfp\" (UniqueName: \"kubernetes.io/projected/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-kube-api-access-n4tfp\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.001990 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.001878 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-ca\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.001990 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.001902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqfk\" (UniqueName: \"kubernetes.io/projected/5b8f8e78-eeb6-48f1-8141-09230494dc60-kube-api-access-cwqfk\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.002118 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.001990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.002118 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.002043 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5b8f8e78-eeb6-48f1-8141-09230494dc60-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.002118 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.002067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.002118 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.002084 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8f8e78-eeb6-48f1-8141-09230494dc60-tmp\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.002234 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.002133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-hub\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.050979 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.050943 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:31.103392 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103362 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-hub\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.103392 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103393 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f91b21d-a7d3-4670-8701-8d694f84394a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht\" (UID: \"8f91b21d-a7d3-4670-8701-8d694f84394a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:31.103625 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.103625 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103445 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clfv\" (UniqueName: \"kubernetes.io/projected/8f91b21d-a7d3-4670-8701-8d694f84394a-kube-api-access-4clfv\") pod \"managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht\" (UID: \"8f91b21d-a7d3-4670-8701-8d694f84394a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:31.103733 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103625 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4tfp\" (UniqueName: \"kubernetes.io/projected/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-kube-api-access-n4tfp\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.103733 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-ca\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.103881 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqfk\" (UniqueName: \"kubernetes.io/projected/5b8f8e78-eeb6-48f1-8141-09230494dc60-kube-api-access-cwqfk\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.103881 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103808 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.104016 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5b8f8e78-eeb6-48f1-8141-09230494dc60-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.104016 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103956 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.104016 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.103980 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8f8e78-eeb6-48f1-8141-09230494dc60-tmp\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.104795 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.104762 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.104946 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.104764 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b8f8e78-eeb6-48f1-8141-09230494dc60-tmp\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.106372 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.106345 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f91b21d-a7d3-4670-8701-8d694f84394a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht\" (UID: \"8f91b21d-a7d3-4670-8701-8d694f84394a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:31.106459 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.106433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.106562 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.106541 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-ca\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.106602 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.106553 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-hub\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.107018 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.106996 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.107118 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.107103 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5b8f8e78-eeb6-48f1-8141-09230494dc60-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.118325 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.118260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqfk\" (UniqueName: \"kubernetes.io/projected/5b8f8e78-eeb6-48f1-8141-09230494dc60-kube-api-access-cwqfk\") pod \"klusterlet-addon-workmgr-6cbf66ffb4-2lz7d\" (UID: \"5b8f8e78-eeb6-48f1-8141-09230494dc60\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.118548 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.118527 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4tfp\" (UniqueName: \"kubernetes.io/projected/4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989-kube-api-access-n4tfp\") pod \"cluster-proxy-proxy-agent-7fbf89c9b8-bnglb\" (UID: \"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.118607 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.118580 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clfv\" (UniqueName: \"kubernetes.io/projected/8f91b21d-a7d3-4670-8701-8d694f84394a-kube-api-access-4clfv\") pod \"managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht\" (UID: \"8f91b21d-a7d3-4670-8701-8d694f84394a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:31.152547 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.152521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:31.172413 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.172390 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:31.198619 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.198585 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" Sep 10 02:34:31.201715 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.201691 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" Sep 10 02:34:31.270561 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.270527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:31.355896 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.355860 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:31.424067 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.423998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:31.446604 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.446578 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d"] Sep 10 02:34:31.449705 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:31.449682 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8f8e78_eeb6_48f1_8141_09230494dc60.slice/crio-1bfb357ed5d9850795a9b91186be59eb696508c195f29235209bc00cb7ebf3d4 WatchSource:0}: Error finding container 1bfb357ed5d9850795a9b91186be59eb696508c195f29235209bc00cb7ebf3d4: Status 404 returned error can't find the container with id 1bfb357ed5d9850795a9b91186be59eb696508c195f29235209bc00cb7ebf3d4 Sep 10 02:34:31.489072 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.489052 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht"] Sep 10 02:34:31.491730 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:31.491693 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f91b21d_a7d3_4670_8701_8d694f84394a.slice/crio-e091c560bc46f1d4ddc60e672e467d24b716f625da94921dd847f87897172102 WatchSource:0}: Error finding container e091c560bc46f1d4ddc60e672e467d24b716f625da94921dd847f87897172102: Status 404 returned error can't find the container with id e091c560bc46f1d4ddc60e672e467d24b716f625da94921dd847f87897172102 Sep 10 02:34:31.492562 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.492538 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb"] Sep 10 02:34:31.495168 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:31.495149 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b7fedcb_0a2f_4b9f_adc1_c496c1f5a989.slice/crio-c331672afd3d59aba5bda7ecbebe2abf45101aec74d67b802c0435cdfbe67b6f WatchSource:0}: Error finding container c331672afd3d59aba5bda7ecbebe2abf45101aec74d67b802c0435cdfbe67b6f: Status 404 returned error can't find the container with id c331672afd3d59aba5bda7ecbebe2abf45101aec74d67b802c0435cdfbe67b6f Sep 10 02:34:31.661096 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.661061 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" event={"ID":"8f91b21d-a7d3-4670-8701-8d694f84394a","Type":"ContainerStarted","Data":"e091c560bc46f1d4ddc60e672e467d24b716f625da94921dd847f87897172102"} Sep 10 02:34:31.662123 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.662097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" event={"ID":"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989","Type":"ContainerStarted","Data":"c331672afd3d59aba5bda7ecbebe2abf45101aec74d67b802c0435cdfbe67b6f"} Sep 10 02:34:31.663045 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:31.663029 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" event={"ID":"5b8f8e78-eeb6-48f1-8141-09230494dc60","Type":"ContainerStarted","Data":"1bfb357ed5d9850795a9b91186be59eb696508c195f29235209bc00cb7ebf3d4"} Sep 10 02:34:32.611725 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:32.610072 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:32.611725 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:32.610608 2570 scope.go:117] "RemoveContainer" containerID="668d9682183bf853bf06c5cd9e974c5d3db4c1d928f48865e633858eef31f2e3" Sep 10 02:34:32.611725 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:32.610848 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-xr5ft_openshift-console-operator(a2400c62-09d7-4e9a-9640-85ce68a09fb4)\"" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" podUID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" Sep 10 02:34:32.611725 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:32.611656 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:32.669523 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:32.668922 2570 scope.go:117] "RemoveContainer" containerID="668d9682183bf853bf06c5cd9e974c5d3db4c1d928f48865e633858eef31f2e3" Sep 10 02:34:32.669523 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:32.669141 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-xr5ft_openshift-console-operator(a2400c62-09d7-4e9a-9640-85ce68a09fb4)\"" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" podUID="a2400c62-09d7-4e9a-9640-85ce68a09fb4" Sep 10 02:34:33.693998 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:33.693855 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:33.789464 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:33.789430 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:33.847542 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:33.847453 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:34.141849 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.141329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:34.146136 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.146086 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ffbf8b54-3714-486c-bc13-4eeb9cd28d26-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-frrmp\" (UID: \"ffbf8b54-3714-486c-bc13-4eeb9cd28d26\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:34.242867 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.242807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:34.243047 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.242897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:34.243768 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.243745 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d22aa7ab-c491-460d-8415-d452a6a64976-service-ca-bundle\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:34.245723 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.245700 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d22aa7ab-c491-460d-8415-d452a6a64976-metrics-certs\") pod \"router-default-6fb5dd5bbd-khhbd\" (UID: \"d22aa7ab-c491-460d-8415-d452a6a64976\") " pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:34.323466 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.323430 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8vbjv\"" Sep 10 02:34:34.331434 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.331402 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" Sep 10 02:34:34.434613 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.434442 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-px9cc\"" Sep 10 02:34:34.442927 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:34.442622 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:36.567930 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:36.567886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:36.570971 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:36.570924 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9f5105d-d773-45f9-afc2-69a1f0fbc218-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-99qz4\" (UID: \"a9f5105d-d773-45f9-afc2-69a1f0fbc218\") " pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:36.708132 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:36.708088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:36.757558 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:36.757525 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jml7x\"" Sep 10 02:34:36.765056 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:36.765027 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" Sep 10 02:34:36.768799 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:36.768678 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:37.895987 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:37.895874 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp"] Sep 10 02:34:37.898502 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:37.898477 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbf8b54_3714_486c_bc13_4eeb9cd28d26.slice/crio-e8d0eeabe7cc98605e73b2f9f70b9c5bc65618c25e92f93b3f1d75d0e69d1629 WatchSource:0}: Error finding container e8d0eeabe7cc98605e73b2f9f70b9c5bc65618c25e92f93b3f1d75d0e69d1629: Status 404 returned error can't find the container with id e8d0eeabe7cc98605e73b2f9f70b9c5bc65618c25e92f93b3f1d75d0e69d1629 Sep 10 02:34:37.901414 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:37.901355 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fb5dd5bbd-khhbd"] Sep 10 02:34:37.904048 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:37.904024 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22aa7ab_c491_460d_8415_d452a6a64976.slice/crio-f5775281b795157beea22ca4cb18ae3ff2da1f3687ef895086f7e148ea9ee7be WatchSource:0}: Error finding container f5775281b795157beea22ca4cb18ae3ff2da1f3687ef895086f7e148ea9ee7be: Status 404 returned error can't find the container with id f5775281b795157beea22ca4cb18ae3ff2da1f3687ef895086f7e148ea9ee7be Sep 10 02:34:37.904188 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:37.904168 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-99qz4"] Sep 10 02:34:37.907428 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:37.907410 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f5105d_d773_45f9_afc2_69a1f0fbc218.slice/crio-6d07649379db7e29330df3e2319ebd874f2de85ba76e15eca70bf7e41dc68cb8 WatchSource:0}: Error finding container 6d07649379db7e29330df3e2319ebd874f2de85ba76e15eca70bf7e41dc68cb8: Status 404 returned error can't find the container with id 6d07649379db7e29330df3e2319ebd874f2de85ba76e15eca70bf7e41dc68cb8 Sep 10 02:34:38.699910 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.699868 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" event={"ID":"ffbf8b54-3714-486c-bc13-4eeb9cd28d26","Type":"ContainerStarted","Data":"e8d0eeabe7cc98605e73b2f9f70b9c5bc65618c25e92f93b3f1d75d0e69d1629"} Sep 10 02:34:38.701639 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.701610 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" event={"ID":"d22aa7ab-c491-460d-8415-d452a6a64976","Type":"ContainerStarted","Data":"178be0ec75e0586693b65320b5ea4e1f54bdece9f08d944b9f6ac24664d6fc93"} Sep 10 02:34:38.701757 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.701655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" event={"ID":"d22aa7ab-c491-460d-8415-d452a6a64976","Type":"ContainerStarted","Data":"f5775281b795157beea22ca4cb18ae3ff2da1f3687ef895086f7e148ea9ee7be"} Sep 10 02:34:38.704773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.704738 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" event={"ID":"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989","Type":"ContainerStarted","Data":"eb73de290cbc0bded7e758f1ab69a645ceb74f08859225c61a722c69720048d1"} Sep 10 02:34:38.704891 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.704782 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" event={"ID":"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989","Type":"ContainerStarted","Data":"8f112a6116b59a814d678eefffcebb4722644e4384362e47f59b1672bb017f68"} Sep 10 02:34:38.704891 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.704801 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" event={"ID":"4b7fedcb-0a2f-4b9f-adc1-c496c1f5a989","Type":"ContainerStarted","Data":"23fb65fff0e6d384b94c8cf6a978f1780fd8a098adbb2ed2065716a5176b4a97"} Sep 10 02:34:38.707125 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.707094 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" event={"ID":"a9f5105d-d773-45f9-afc2-69a1f0fbc218","Type":"ContainerStarted","Data":"6d07649379db7e29330df3e2319ebd874f2de85ba76e15eca70bf7e41dc68cb8"} Sep 10 02:34:38.708799 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.708779 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" event={"ID":"8f91b21d-a7d3-4670-8701-8d694f84394a","Type":"ContainerStarted","Data":"a2a0161639f82216f375fce108e52f3fd02fbd0549a9524bc5c9983150275496"} Sep 10 02:34:38.710514 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.710490 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" event={"ID":"5b8f8e78-eeb6-48f1-8141-09230494dc60","Type":"ContainerStarted","Data":"ce86b053a5eaffa57c10b82902ae7c7af4a48ee3f3f57435598de18f791a36a6"} Sep 10 02:34:38.710706 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.710688 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:38.712214 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.712194 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" Sep 10 02:34:38.726314 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.726267 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" podStartSLOduration=36.726249631 podStartE2EDuration="36.726249631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:38.72467395 +0000 UTC m=+87.990770342" watchObservedRunningTime="2026-09-10 02:34:38.726249631 +0000 UTC m=+87.992346023" Sep 10 02:34:38.744496 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.744448 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cbf66ffb4-2lz7d" podStartSLOduration=2.572335019 podStartE2EDuration="8.744430408s" totalImagesPullingTime="6.172095389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:38.742994776 +0000 UTC m=+88.009091169" watchObservedRunningTime="2026-09-10 02:34:38.744430408 +0000 UTC m=+88.010526779" Sep 10 02:34:38.761375 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.761331 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fc9f4f7b-g4tht" podStartSLOduration=2.621104763 podStartE2EDuration="8.761314085s" totalImagesPullingTime="6.140209322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:38.759793691 +0000 UTC m=+88.025890082" watchObservedRunningTime="2026-09-10 02:34:38.761314085 +0000 UTC m=+88.027410476" Sep 10 02:34:38.783506 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:38.783332 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7fbf89c9b8-bnglb" podStartSLOduration=2.648036432 podStartE2EDuration="8.783317082s" totalImagesPullingTime="6.13528065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:38.783199309 +0000 UTC m=+88.049295701" watchObservedRunningTime="2026-09-10 02:34:38.783317082 +0000 UTC m=+88.049413473" Sep 10 02:34:39.443918 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:39.443882 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:39.446959 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:39.446934 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:39.739723 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:39.739703 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:39.741030 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:39.741013 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6fb5dd5bbd-khhbd" Sep 10 02:34:40.412600 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.412524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:40.511098 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.511068 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:40.578126 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.578093 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:40.706800 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.706770 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:40.719097 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.719068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" event={"ID":"a9f5105d-d773-45f9-afc2-69a1f0fbc218","Type":"ContainerStarted","Data":"06e9eb79c4ffbf701a0823c458c1adcc106786316384064eb16cf902683d352f"} Sep 10 02:34:40.720617 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.720596 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" event={"ID":"ffbf8b54-3714-486c-bc13-4eeb9cd28d26","Type":"ContainerStarted","Data":"c03f4ce2724053da34b5647fc58e932bde8ff3bdf7d7458943b104ba4979aa5c"} Sep 10 02:34:40.733872 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.733805 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-99qz4" podStartSLOduration=34.940503136 podStartE2EDuration="36.733792529s" totalImagesPullingTime="1.793289393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:40.733278718 +0000 UTC m=+89.999375108" watchObservedRunningTime="2026-09-10 02:34:40.733792529 +0000 UTC m=+89.999888918" Sep 10 02:34:40.752762 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.752720 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-frrmp" podStartSLOduration=36.594883642 podStartE2EDuration="38.752711635s" totalImagesPullingTime="2.157827993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:40.752209417 +0000 UTC m=+90.018305807" watchObservedRunningTime="2026-09-10 02:34:40.752711635 +0000 UTC m=+90.018808025" Sep 10 02:34:40.841690 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.841656 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:40.900233 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:40.900200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:41.311022 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:41.310982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:41.411371 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:41.411334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:44.599094 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:44.599051 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:44.661513 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:44.661475 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:45.222338 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:45.222307 2570 scope.go:117] "RemoveContainer" containerID="668d9682183bf853bf06c5cd9e974c5d3db4c1d928f48865e633858eef31f2e3" Sep 10 02:34:45.279259 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:45.279229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:45.653266 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:45.653171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:45.744992 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:45.744964 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:34:45.745140 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:45.745069 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" event={"ID":"a2400c62-09d7-4e9a-9640-85ce68a09fb4","Type":"ContainerStarted","Data":"54f98de519fe5c7740db0e08cb2839ab5a2270f6b4db5288a7c8eb431d24f464"} Sep 10 02:34:45.745452 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:45.745420 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:45.762344 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:45.762298 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" podStartSLOduration=38.720716655 podStartE2EDuration="43.762279155s" totalImagesPullingTime="5.0415625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:45.760950496 +0000 UTC m=+95.027046890" watchObservedRunningTime="2026-09-10 02:34:45.762279155 +0000 UTC m=+95.028375549" Sep 10 02:34:46.226240 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:46.226212 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-xr5ft" Sep 10 02:34:46.435077 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:46.435043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:46.495171 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:46.495086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:46.596856 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:46.596809 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:46.701626 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:46.701595 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:46.890176 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:46.890094 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:46.947211 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:46.947180 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:47.278926 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:47.278889 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:47.378490 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:47.378459 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:47.482514 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:47.482484 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:47.677922 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:47.677886 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:47.778747 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:47.778713 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:47.989385 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:47.989286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:48.090188 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:48.090159 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:48.190871 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:48.190821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:48.296075 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:48.295982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:48.392951 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:48.392918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:48.493027 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:48.492993 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:48.593992 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:48.593918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:49.578874 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.578783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:34:49.578874 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.578820 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:34:49.581050 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.581031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"image-registry-5b47995f-scq9k\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:34:49.581142 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.581030 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d322117c-a2b5-4598-8c62-31ab1911f3cd-metrics-tls\") pod \"dns-default-v8xtg\" (UID: \"d322117c-a2b5-4598-8c62-31ab1911f3cd\") " pod="openshift-dns/dns-default-v8xtg" Sep 10 02:34:49.607055 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.607035 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9rzqt" Sep 10 02:34:49.708094 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.708059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:49.799792 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.799765 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pb55z\"" Sep 10 02:34:49.807657 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.807634 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:34:49.807784 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.807721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:49.814361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.814344 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4fcfm\"" Sep 10 02:34:49.822732 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:49.822709 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v8xtg" Sep 10 02:34:50.084004 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.083928 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b47995f-scq9k"] Sep 10 02:34:50.087134 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:50.087110 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7a6fc4_786a_4e2e_84d9_283835e132ed.slice/crio-6703b89dfe00489152b9bc021273efdb5aee546bcd0160feccf2edca39432863 WatchSource:0}: Error finding container 6703b89dfe00489152b9bc021273efdb5aee546bcd0160feccf2edca39432863: Status 404 returned error can't find the container with id 6703b89dfe00489152b9bc021273efdb5aee546bcd0160feccf2edca39432863 Sep 10 02:34:50.107664 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.105413 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v8xtg"] Sep 10 02:34:50.117788 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:50.117767 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd322117c_a2b5_4598_8c62_31ab1911f3cd.slice/crio-5d787b419c5a1a38fb86cd16822ec9c90416c84ec7eef7b1f89257af676c5dc2 WatchSource:0}: Error finding container 5d787b419c5a1a38fb86cd16822ec9c90416c84ec7eef7b1f89257af676c5dc2: Status 404 returned error can't find the container with id 5d787b419c5a1a38fb86cd16822ec9c90416c84ec7eef7b1f89257af676c5dc2 Sep 10 02:34:50.716917 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.716881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:50.764458 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.764429 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b47995f-scq9k" event={"ID":"dc7a6fc4-786a-4e2e-84d9-283835e132ed","Type":"ContainerStarted","Data":"ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed"} Sep 10 02:34:50.764607 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.764479 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b47995f-scq9k" event={"ID":"dc7a6fc4-786a-4e2e-84d9-283835e132ed","Type":"ContainerStarted","Data":"6703b89dfe00489152b9bc021273efdb5aee546bcd0160feccf2edca39432863"} Sep 10 02:34:50.764607 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.764530 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:34:50.765860 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.765819 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8xtg" event={"ID":"d322117c-a2b5-4598-8c62-31ab1911f3cd","Type":"ContainerStarted","Data":"5d787b419c5a1a38fb86cd16822ec9c90416c84ec7eef7b1f89257af676c5dc2"} Sep 10 02:34:50.790686 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.790636 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b47995f-scq9k" podStartSLOduration=78.790617199 podStartE2EDuration="1m18.790617199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:50.790595196 +0000 UTC m=+100.056691599" watchObservedRunningTime="2026-09-10 02:34:50.790617199 +0000 UTC m=+100.056713591" Sep 10 02:34:50.822548 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.822468 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:50.925749 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:50.925650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.004256 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.003912 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dgrfb"] Sep 10 02:34:51.007888 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.007857 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.010598 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.010568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 02:34:51.011150 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.010994 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 02:34:51.012065 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.011716 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qwtm5\"" Sep 10 02:34:51.012065 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.011761 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 02:34:51.012065 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.011926 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 02:34:51.049738 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.049715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096148 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjx6t\" (UniqueName: \"kubernetes.io/projected/b18c3d5a-7b08-4dc2-88a1-a3e576394832-kube-api-access-jjx6t\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096276 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-textfile\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096343 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b18c3d5a-7b08-4dc2-88a1-a3e576394832-metrics-client-ca\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-wtmp\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096574 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-sys\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096647 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.096840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096721 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-tls\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.100280 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.096752 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-root\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.139816 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.139788 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198549 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-wtmp\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-sys\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198613 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-tls\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-root\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198714 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjx6t\" (UniqueName: \"kubernetes.io/projected/b18c3d5a-7b08-4dc2-88a1-a3e576394832-kube-api-access-jjx6t\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-textfile\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.198843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.198783 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b18c3d5a-7b08-4dc2-88a1-a3e576394832-metrics-client-ca\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.199369 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:51.198967 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 02:34:51.199369 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:34:51.199031 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-tls podName:b18c3d5a-7b08-4dc2-88a1-a3e576394832 nodeName:}" failed. No retries permitted until 2026-09-10 02:34:51.699013102 +0000 UTC m=+100.965109479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-tls") pod "node-exporter-dgrfb" (UID: "b18c3d5a-7b08-4dc2-88a1-a3e576394832") : secret "node-exporter-tls" not found Sep 10 02:34:51.199369 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.199122 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-accelerators-collector-config\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.199369 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.199179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-sys\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.199369 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.199339 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-wtmp\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.199619 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.199385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b18c3d5a-7b08-4dc2-88a1-a3e576394832-metrics-client-ca\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.199673 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.199615 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-textfile\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.199673 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.199626 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b18c3d5a-7b08-4dc2-88a1-a3e576394832-root\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.208846 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.207471 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.213332 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.213285 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjx6t\" (UniqueName: \"kubernetes.io/projected/b18c3d5a-7b08-4dc2-88a1-a3e576394832-kube-api-access-jjx6t\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.241285 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.241251 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.347553 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.347196 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.440018 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.439980 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.541758 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.541717 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.703708 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.703676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-tls\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.706299 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.706277 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b18c3d5a-7b08-4dc2-88a1-a3e576394832-node-exporter-tls\") pod \"node-exporter-dgrfb\" (UID: \"b18c3d5a-7b08-4dc2-88a1-a3e576394832\") " pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.745472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.745154 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:51.921930 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.921903 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dgrfb" Sep 10 02:34:51.944232 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:34:51.944199 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18c3d5a_7b08_4dc2_88a1_a3e576394832.slice/crio-4b7c0ad8f8ec478fba09bc2de066a07a2da4e45da196241a13cd49d2c556cc22 WatchSource:0}: Error finding container 4b7c0ad8f8ec478fba09bc2de066a07a2da4e45da196241a13cd49d2c556cc22: Status 404 returned error can't find the container with id 4b7c0ad8f8ec478fba09bc2de066a07a2da4e45da196241a13cd49d2c556cc22 Sep 10 02:34:51.959022 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:51.958686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:52.047521 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:52.047423 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:52.166802 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:52.166768 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:52.776685 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:52.776641 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgrfb" event={"ID":"b18c3d5a-7b08-4dc2-88a1-a3e576394832","Type":"ContainerStarted","Data":"4b7c0ad8f8ec478fba09bc2de066a07a2da4e45da196241a13cd49d2c556cc22"} Sep 10 02:34:52.778759 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:52.778731 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8xtg" event={"ID":"d322117c-a2b5-4598-8c62-31ab1911f3cd","Type":"ContainerStarted","Data":"7d12cff7a93fbffb603fc0a4b5d622b649e5e27cbe73709f9160650550af042e"} Sep 10 02:34:52.778899 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:52.778779 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v8xtg" event={"ID":"d322117c-a2b5-4598-8c62-31ab1911f3cd","Type":"ContainerStarted","Data":"c3bfc0b0eb1ef7dae14e8a948242ed53bb52cb56907aeb6bb7c50f4830ac30fb"} Sep 10 02:34:52.778899 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:52.778870 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v8xtg" Sep 10 02:34:52.800309 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:52.800268 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v8xtg" podStartSLOduration=66.018116357 podStartE2EDuration="1m7.800249427s" totalImagesPullingTime="1.78213307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:52.798610794 +0000 UTC m=+102.064707185" watchObservedRunningTime="2026-09-10 02:34:52.800249427 +0000 UTC m=+102.066345816" Sep 10 02:34:53.274073 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.274020 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b47995f-scq9k"] Sep 10 02:34:53.354885 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.354795 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:53.638971 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.638888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:53.740507 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.740469 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:53.785157 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.785128 2570 generic.go:356] "Generic (PLEG): container finished" podID="b18c3d5a-7b08-4dc2-88a1-a3e576394832" containerID="b4157b550c0f79c3c01a4a8e26a2add676d4e30242702be4e13ff43616d73914" exitCode=0 Sep 10 02:34:53.785545 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.785220 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgrfb" event={"ID":"b18c3d5a-7b08-4dc2-88a1-a3e576394832","Type":"ContainerDied","Data":"b4157b550c0f79c3c01a4a8e26a2add676d4e30242702be4e13ff43616d73914"} Sep 10 02:34:53.880015 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.879973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:53.982122 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:53.982087 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.161269 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.161181 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.214396 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.214363 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.372893 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.372851 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.472352 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.472313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.646864 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.646809 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.709843 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.709795 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.775462 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.775387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:54.792102 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.792074 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgrfb" event={"ID":"b18c3d5a-7b08-4dc2-88a1-a3e576394832","Type":"ContainerStarted","Data":"b3da42dbf3e92a434bb54593e52464f312c28aae7f452313d9f8ffe2b59bf249"} Sep 10 02:34:54.792472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.792125 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dgrfb" event={"ID":"b18c3d5a-7b08-4dc2-88a1-a3e576394832","Type":"ContainerStarted","Data":"242c4bce5806974549550a0ac48f57a5aedf32c3501d00943f08b6c44c2574db"} Sep 10 02:34:54.812916 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:54.812871 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dgrfb" podStartSLOduration=3.771144136 podStartE2EDuration="4.812857718s" totalImagesPullingTime="982.084503ms" totalInitContainerRuntime="59.629079ms" isStatefulPod=true podCreationTimestamp="2026-09-10 02:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:34:54.81174138 +0000 UTC m=+104.077837770" watchObservedRunningTime="2026-09-10 02:34:54.812857718 +0000 UTC m=+104.078954132" Sep 10 02:34:55.733095 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:55.733062 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:55.831994 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:55.831958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:56.003755 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:56.003673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:57.181599 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:57.181561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:59.794512 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:59.794483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:34:59.849682 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:34:59.849634 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:01.314318 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:01.314277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:01.417512 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:01.417480 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:02.787627 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:02.787591 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v8xtg" Sep 10 02:35:02.909802 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:02.909762 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:03.010454 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:03.010402 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:05.283733 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:05.283697 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:05.332909 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:05.332876 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:07.106975 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:07.106938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:07.162937 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:07.162895 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:08.964214 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:08.964172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:09.070268 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:09.070233 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:09.185969 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:09.185929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:09.284125 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:09.284040 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:12.370523 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:12.370482 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:12.423910 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:12.423872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:13.280316 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:13.280285 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:35:14.359362 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:14.359329 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:14.459253 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:14.459212 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:14.802975 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:14.802938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:14.904583 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:14.904554 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:18.293278 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.293229 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b47995f-scq9k" podUID="dc7a6fc4-786a-4e2e-84d9-283835e132ed" containerName="registry" containerID="cri-o://ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed" gracePeriod=30 Sep 10 02:35:18.545943 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.545887 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:35:18.630336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630299 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-trusted-ca\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630527 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630353 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-installation-pull-secrets\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630527 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630373 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-image-registry-private-configuration\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630527 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630394 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-certificates\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630527 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630438 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630527 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630464 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-bound-sa-token\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630527 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630514 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql4mj\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-kube-api-access-ql4mj\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630853 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630568 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc7a6fc4-786a-4e2e-84d9-283835e132ed-ca-trust-extracted\") pod \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\" (UID: \"dc7a6fc4-786a-4e2e-84d9-283835e132ed\") " Sep 10 02:35:18.630927 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630811 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:35:18.630927 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.630890 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:35:18.633110 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.633076 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-kube-api-access-ql4mj" (OuterVolumeSpecName: "kube-api-access-ql4mj") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "kube-api-access-ql4mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:35:18.633224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.633151 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:35:18.633224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.633169 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:35:18.633224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.633189 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:35:18.633325 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.633228 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:35:18.638895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.638872 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7a6fc4-786a-4e2e-84d9-283835e132ed-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dc7a6fc4-786a-4e2e-84d9-283835e132ed" (UID: "dc7a6fc4-786a-4e2e-84d9-283835e132ed"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 02:35:18.731531 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731479 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-tls\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.731531 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731528 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-bound-sa-token\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.731531 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731541 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql4mj\" (UniqueName: \"kubernetes.io/projected/dc7a6fc4-786a-4e2e-84d9-283835e132ed-kube-api-access-ql4mj\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.731773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731554 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc7a6fc4-786a-4e2e-84d9-283835e132ed-ca-trust-extracted\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.731773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731563 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-trusted-ca\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.731773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731576 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-installation-pull-secrets\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.731773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731585 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc7a6fc4-786a-4e2e-84d9-283835e132ed-image-registry-private-configuration\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.731773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.731594 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc7a6fc4-786a-4e2e-84d9-283835e132ed-registry-certificates\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:35:18.881628 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.881541 2570 generic.go:356] "Generic (PLEG): container finished" podID="dc7a6fc4-786a-4e2e-84d9-283835e132ed" containerID="ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed" exitCode=0 Sep 10 02:35:18.881628 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.881584 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b47995f-scq9k" Sep 10 02:35:18.881845 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.881632 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b47995f-scq9k" event={"ID":"dc7a6fc4-786a-4e2e-84d9-283835e132ed","Type":"ContainerDied","Data":"ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed"} Sep 10 02:35:18.881845 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.881681 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b47995f-scq9k" event={"ID":"dc7a6fc4-786a-4e2e-84d9-283835e132ed","Type":"ContainerDied","Data":"6703b89dfe00489152b9bc021273efdb5aee546bcd0160feccf2edca39432863"} Sep 10 02:35:18.881845 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.881701 2570 scope.go:117] "RemoveContainer" containerID="ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed" Sep 10 02:35:18.883197 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.883174 2570 generic.go:356] "Generic (PLEG): container finished" podID="92db0478-ff93-4fb6-a696-0885419be325" containerID="dad8dd141e6f4e95a591804f52245efc7ed33bdc4e3793a7f6c6b8cf82927a8d" exitCode=0 Sep 10 02:35:18.883280 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.883209 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" event={"ID":"92db0478-ff93-4fb6-a696-0885419be325","Type":"ContainerDied","Data":"dad8dd141e6f4e95a591804f52245efc7ed33bdc4e3793a7f6c6b8cf82927a8d"} Sep 10 02:35:18.883590 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.883574 2570 scope.go:117] "RemoveContainer" containerID="dad8dd141e6f4e95a591804f52245efc7ed33bdc4e3793a7f6c6b8cf82927a8d" Sep 10 02:35:18.892812 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.892795 2570 scope.go:117] "RemoveContainer" containerID="ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed" Sep 10 02:35:18.893079 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:35:18.893062 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed\": container with ID starting with ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed not found: ID does not exist" containerID="ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed" Sep 10 02:35:18.893138 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.893085 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed"} err="failed to get container status \"ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed\": rpc error: code = NotFound desc = could not find container \"ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed\": container with ID starting with ef6d8dc9513f1fc1eb211dba2c8b12b4ede64ce13b5749b4e264f86f9e10feed not found: ID does not exist" Sep 10 02:35:18.918046 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.918003 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b47995f-scq9k"] Sep 10 02:35:18.924297 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:18.924270 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b47995f-scq9k"] Sep 10 02:35:19.227553 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:19.227520 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc7a6fc4-786a-4e2e-84d9-283835e132ed" path="/var/lib/kubelet/pods/dc7a6fc4-786a-4e2e-84d9-283835e132ed/volumes" Sep 10 02:35:19.890515 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:19.890477 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-bk7gd" event={"ID":"92db0478-ff93-4fb6-a696-0885419be325","Type":"ContainerStarted","Data":"b6fab8a687610ed0f0157574c0516610e8e158fc932f14600e711f447d593883"} Sep 10 02:35:21.050716 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:21.050683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:35:21.052983 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:21.052956 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253d3186-27d1-4b7e-b48a-303871be8896-metrics-certs\") pod \"network-metrics-daemon-6hhxs\" (UID: \"253d3186-27d1-4b7e-b48a-303871be8896\") " pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:35:21.086612 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:21.086577 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-frrmp_ffbf8b54-3714-486c-bc13-4eeb9cd28d26/cluster-monitoring-operator/0.log" Sep 10 02:35:21.342421 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:21.342338 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwtct\"" Sep 10 02:35:21.350725 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:21.350694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhxs" Sep 10 02:35:21.620892 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:21.620801 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6hhxs"] Sep 10 02:35:21.623480 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:35:21.623452 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253d3186_27d1_4b7e_b48a_303871be8896.slice/crio-d2c2fa9170537fca93ee9b2d10796ee56c46a128d13733350b995db4f6a23e91 WatchSource:0}: Error finding container d2c2fa9170537fca93ee9b2d10796ee56c46a128d13733350b995db4f6a23e91: Status 404 returned error can't find the container with id d2c2fa9170537fca93ee9b2d10796ee56c46a128d13733350b995db4f6a23e91 Sep 10 02:35:21.900156 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:21.900072 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hhxs" event={"ID":"253d3186-27d1-4b7e-b48a-303871be8896","Type":"ContainerStarted","Data":"d2c2fa9170537fca93ee9b2d10796ee56c46a128d13733350b995db4f6a23e91"} Sep 10 02:35:22.880858 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:22.880754 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgrfb_b18c3d5a-7b08-4dc2-88a1-a3e576394832/init-textfile/0.log" Sep 10 02:35:22.905986 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:22.905950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hhxs" event={"ID":"253d3186-27d1-4b7e-b48a-303871be8896","Type":"ContainerStarted","Data":"d62cdfeaf635a07daa5e0de63970a0d6667d7e71eca2de9e162c08f6b1d2755a"} Sep 10 02:35:22.906159 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:22.906003 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hhxs" event={"ID":"253d3186-27d1-4b7e-b48a-303871be8896","Type":"ContainerStarted","Data":"7cca6f4ee29288efb22496cafe06393bfdaedf8f97a5d36bc41beedd3c9c75be"} Sep 10 02:35:22.923380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:22.923325 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6hhxs" podStartSLOduration=130.941958185 podStartE2EDuration="2m11.923306281s" totalImagesPullingTime="981.348096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:35:22.921181726 +0000 UTC m=+132.187278120" watchObservedRunningTime="2026-09-10 02:35:22.923306281 +0000 UTC m=+132.189402672" Sep 10 02:35:23.006408 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.006370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:23.081573 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.081533 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgrfb_b18c3d5a-7b08-4dc2-88a1-a3e576394832/node-exporter/0.log" Sep 10 02:35:23.110057 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.110006 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:23.164087 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.163988 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:23.280916 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.280888 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgrfb_b18c3d5a-7b08-4dc2-88a1-a3e576394832/kube-rbac-proxy/0.log" Sep 10 02:35:23.911273 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.911240 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ea64165-5263-4e39-b47c-d69201ac3a2b" containerID="e0a9e5e5b432397f3bf05d14aa21b3912c36c9572a8a966412d46961c8a0b2f5" exitCode=0 Sep 10 02:35:23.911672 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.911320 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" event={"ID":"6ea64165-5263-4e39-b47c-d69201ac3a2b","Type":"ContainerDied","Data":"e0a9e5e5b432397f3bf05d14aa21b3912c36c9572a8a966412d46961c8a0b2f5"} Sep 10 02:35:23.911752 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:23.911737 2570 scope.go:117] "RemoveContainer" containerID="e0a9e5e5b432397f3bf05d14aa21b3912c36c9572a8a966412d46961c8a0b2f5" Sep 10 02:35:24.139668 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:24.139568 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:24.241504 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:24.241463 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:24.913146 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:24.913106 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:24.921622 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:24.921591 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-ngs6c" event={"ID":"6ea64165-5263-4e39-b47c-d69201ac3a2b","Type":"ContainerStarted","Data":"0faebded165da9cf015218b98a425d5577eb9c96e6e68f8692d3b57a2f9bcfe7"} Sep 10 02:35:25.014584 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:25.014534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:28.480881 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:28.480849 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-99qz4_a9f5105d-d773-45f9-afc2-69a1f0fbc218/networking-console-plugin/0.log" Sep 10 02:35:28.680574 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:28.680545 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:35:28.886038 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:28.885958 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/3.log" Sep 10 02:35:29.808721 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:29.808686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:29.909505 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:29.909470 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:38.981300 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:38.981262 2570 generic.go:356] "Generic (PLEG): container finished" podID="24a99e6f-fe48-48fc-a340-09ae9d1728fc" containerID="f2b162a1188606425646050887eea05980aee3b61ff3d4aed514928861d8c32f" exitCode=0 Sep 10 02:35:38.981688 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:38.981334 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" event={"ID":"24a99e6f-fe48-48fc-a340-09ae9d1728fc","Type":"ContainerDied","Data":"f2b162a1188606425646050887eea05980aee3b61ff3d4aed514928861d8c32f"} Sep 10 02:35:38.981688 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:38.981665 2570 scope.go:117] "RemoveContainer" containerID="f2b162a1188606425646050887eea05980aee3b61ff3d4aed514928861d8c32f" Sep 10 02:35:39.987260 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:39.987225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-szhxg" event={"ID":"24a99e6f-fe48-48fc-a340-09ae9d1728fc","Type":"ContainerStarted","Data":"9e7db9b25f1981470c6719193e89e1954394dabad53118fdbe14736c5bdab0bd"} Sep 10 02:35:44.820902 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:44.820858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:35:44.894365 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:35:44.894324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:07.025548 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:07.025470 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:07.088286 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:07.088235 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:14.120541 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:14.120498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:14.180216 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:14.180181 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:18.900394 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:18.900362 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:19.000050 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:19.000013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:24.949149 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:24.949107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:25.047216 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:25.047181 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:25.109366 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:25.109324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:29.800972 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:29.800931 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:29.913729 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:29.913689 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:41.012022 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:41.011982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:50.165174 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:50.165125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:36:50.238789 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:36:50.238751 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:37:37.329129 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:37:37.329049 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:37:42.418494 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:37:42.418460 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:37:42.517543 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:37:42.517501 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:37:54.121694 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:37:54.121658 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:37:54.223995 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:37:54.223963 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:09.302112 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:09.302057 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:10.138879 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:10.138808 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:11.159367 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:11.159336 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:38:11.166689 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:11.166653 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:38:11.176469 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:11.176427 2570 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 02:38:13.100838 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:13.100800 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:13.162942 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:13.162907 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:14.681685 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:14.681650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:14.844691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:14.844652 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:14.991757 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:14.991680 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:17.098097 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:17.098056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:17.507086 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:17.507049 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:19.712652 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:19.712613 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:19.813232 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:19.813197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:23.120048 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:23.120012 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:23.221427 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:23.221379 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:24.743594 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:24.743546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:27.220611 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:27.220561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:27.310976 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:27.310940 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:29.293096 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:29.293055 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:29.393061 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:29.393027 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:31.037442 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:31.037376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:48.328934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:48.328890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:49.415498 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.415459 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw"] Sep 10 02:38:49.415914 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.415737 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc7a6fc4-786a-4e2e-84d9-283835e132ed" containerName="registry" Sep 10 02:38:49.415914 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.415749 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc7a6fc4-786a-4e2e-84d9-283835e132ed" containerName="registry" Sep 10 02:38:49.415914 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.415816 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc7a6fc4-786a-4e2e-84d9-283835e132ed" containerName="registry" Sep 10 02:38:49.418665 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.418646 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.426946 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.426922 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 02:38:49.427076 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.427042 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rwkzf\"" Sep 10 02:38:49.427545 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.427532 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 02:38:49.449407 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.449382 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 02:38:49.452524 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.452501 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw"] Sep 10 02:38:49.456316 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.456296 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 02:38:49.520037 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.520005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e73accdd-c852-4fd5-b4b1-747844cd7335-apiservice-cert\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.520233 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.520044 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e73accdd-c852-4fd5-b4b1-747844cd7335-tmp\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.520233 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.520078 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e73accdd-c852-4fd5-b4b1-747844cd7335-manifests\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.520233 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.520166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e73accdd-c852-4fd5-b4b1-747844cd7335-webhook-cert\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.520233 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.520182 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfqh\" (UniqueName: \"kubernetes.io/projected/e73accdd-c852-4fd5-b4b1-747844cd7335-kube-api-access-znfqh\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.621366 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.621280 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e73accdd-c852-4fd5-b4b1-747844cd7335-manifests\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.621366 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.621348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e73accdd-c852-4fd5-b4b1-747844cd7335-webhook-cert\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.621599 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.621369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfqh\" (UniqueName: \"kubernetes.io/projected/e73accdd-c852-4fd5-b4b1-747844cd7335-kube-api-access-znfqh\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.621599 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.621391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e73accdd-c852-4fd5-b4b1-747844cd7335-apiservice-cert\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.621599 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.621413 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e73accdd-c852-4fd5-b4b1-747844cd7335-tmp\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.621758 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.621732 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e73accdd-c852-4fd5-b4b1-747844cd7335-manifests\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.623675 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.623645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e73accdd-c852-4fd5-b4b1-747844cd7335-tmp\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.623896 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.623878 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e73accdd-c852-4fd5-b4b1-747844cd7335-webhook-cert\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.623951 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.623916 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e73accdd-c852-4fd5-b4b1-747844cd7335-apiservice-cert\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.632657 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.632634 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfqh\" (UniqueName: \"kubernetes.io/projected/e73accdd-c852-4fd5-b4b1-747844cd7335-kube-api-access-znfqh\") pod \"opendatahub-operator-controller-manager-647784d6dd-s9mvw\" (UID: \"e73accdd-c852-4fd5-b4b1-747844cd7335\") " pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.653233 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.653197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:49.729334 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.729279 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:49.789257 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.789218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:49.886844 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:49.886734 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.001309 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.001274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.025538 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.025515 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw"] Sep 10 02:38:50.027130 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:38:50.027101 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73accdd_c852_4fd5_b4b1_747844cd7335.slice/crio-554a3e23799908b16e4cfa369deac2ac5546845c2223f7966240af9bcb31867a WatchSource:0}: Error finding container 554a3e23799908b16e4cfa369deac2ac5546845c2223f7966240af9bcb31867a: Status 404 returned error can't find the container with id 554a3e23799908b16e4cfa369deac2ac5546845c2223f7966240af9bcb31867a Sep 10 02:38:50.028844 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.028804 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 02:38:50.126008 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.125969 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.194244 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.194183 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.259756 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.259716 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.346309 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.346277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.451012 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.450927 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.551289 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.551245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:38:50.731679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:50.731576 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" event={"ID":"e73accdd-c852-4fd5-b4b1-747844cd7335","Type":"ContainerStarted","Data":"554a3e23799908b16e4cfa369deac2ac5546845c2223f7966240af9bcb31867a"} Sep 10 02:38:53.746672 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:53.746635 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" event={"ID":"e73accdd-c852-4fd5-b4b1-747844cd7335","Type":"ContainerStarted","Data":"5ce221cc18feae2f5300af1906e3455a88a3dd7f3fd14745ed6a3ef88d095999"} Sep 10 02:38:54.752531 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:54.752491 2570 generic.go:356] "Generic (PLEG): container finished" podID="e73accdd-c852-4fd5-b4b1-747844cd7335" containerID="5ce221cc18feae2f5300af1906e3455a88a3dd7f3fd14745ed6a3ef88d095999" exitCode=0 Sep 10 02:38:54.753028 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:54.752576 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" event={"ID":"e73accdd-c852-4fd5-b4b1-747844cd7335","Type":"ContainerDied","Data":"5ce221cc18feae2f5300af1906e3455a88a3dd7f3fd14745ed6a3ef88d095999"} Sep 10 02:38:55.763894 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:55.763849 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" event={"ID":"e73accdd-c852-4fd5-b4b1-747844cd7335","Type":"ContainerStarted","Data":"394aac8cd6e2d32feff01663f17dc1410e2e845ba707a3ef6c62f635df5a65e3"} Sep 10 02:38:55.764283 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:55.764016 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:38:55.790237 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:38:55.790184 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" podStartSLOduration=2.695721329 podStartE2EDuration="6.790168594s" totalImagesPullingTime="3.093140314s" totalInitContainerRuntime="1.001306951s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:38:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:38:55.788278227 +0000 UTC m=+345.054374636" watchObservedRunningTime="2026-09-10 02:38:55.790168594 +0000 UTC m=+345.056264984" Sep 10 02:39:01.540120 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:01.540059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:01.656164 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:01.656125 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:06.770437 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:06.770361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-647784d6dd-s9mvw" Sep 10 02:39:08.870556 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:08.870521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:09.041280 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:09.041242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:10.730393 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.730351 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:10.773609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.773581 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd"] Sep 10 02:39:10.782122 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.782101 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:10.785051 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.785030 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jjzbt\"" Sep 10 02:39:10.785318 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.785300 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 02:39:10.786016 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.785997 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 02:39:10.791005 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.790984 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd"] Sep 10 02:39:10.889297 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.889263 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tmp\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:10.889492 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.889328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kx6\" (UniqueName: \"kubernetes.io/projected/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-kube-api-access-46kx6\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:10.889492 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.889356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tls-certs\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:10.990262 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.990177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tmp\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:10.990262 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.990226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46kx6\" (UniqueName: \"kubernetes.io/projected/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-kube-api-access-46kx6\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:10.990262 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.990251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tls-certs\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:10.990528 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:39:10.990356 2570 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 10 02:39:10.990528 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:39:10.990415 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tls-certs podName:97bd02c0-6aeb-49c0-ba0c-3f4521345b6e nodeName:}" failed. No retries permitted until 2026-09-10 02:39:11.490394822 +0000 UTC m=+360.756491190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tls-certs") pod "kube-auth-proxy-74d6b6cf86-z6zzd" (UID: "97bd02c0-6aeb-49c0-ba0c-3f4521345b6e") : secret "kube-auth-proxy-tls" not found Sep 10 02:39:10.992528 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:10.992509 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tmp\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:11.002065 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:11.002040 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kx6\" (UniqueName: \"kubernetes.io/projected/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-kube-api-access-46kx6\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:11.177953 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:11.177918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:11.495683 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:11.495634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tls-certs\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:11.497992 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:11.497971 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd02c0-6aeb-49c0-ba0c-3f4521345b6e-tls-certs\") pod \"kube-auth-proxy-74d6b6cf86-z6zzd\" (UID: \"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e\") " pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:11.695335 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:11.695298 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jjzbt\"" Sep 10 02:39:11.703691 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:11.703666 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" Sep 10 02:39:11.976528 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:11.976496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd"] Sep 10 02:39:11.980854 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:39:11.980809 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bd02c0_6aeb_49c0_ba0c_3f4521345b6e.slice/crio-f751c129110fe609e38d48463ab6ce127fccda4450dc38b40c8208e863502358 WatchSource:0}: Error finding container f751c129110fe609e38d48463ab6ce127fccda4450dc38b40c8208e863502358: Status 404 returned error can't find the container with id f751c129110fe609e38d48463ab6ce127fccda4450dc38b40c8208e863502358 Sep 10 02:39:12.337347 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.337311 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:12.356844 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.356801 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d"] Sep 10 02:39:12.361331 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.361314 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.364017 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.363988 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 02:39:12.364144 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.363988 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 02:39:12.364144 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.363988 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 02:39:12.364144 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.363991 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 02:39:12.364144 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.364051 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cg2z8\"" Sep 10 02:39:12.374261 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.374241 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d"] Sep 10 02:39:12.429611 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.429578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:12.504931 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.504898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xkm\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-kube-api-access-99xkm\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.505089 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.504944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.505089 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.505021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.505089 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.505054 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.505089 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.505075 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.505236 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.505095 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dff17f8-501c-4dea-a242-8116956effe6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.505236 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.505111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.605752 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.605665 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.605752 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.605730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.605987 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.605762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.605987 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.605790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dff17f8-501c-4dea-a242-8116956effe6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.605987 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.605814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.605987 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.605883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xkm\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-kube-api-access-99xkm\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.605987 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.605931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.606927 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.606876 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.609171 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.609125 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dff17f8-501c-4dea-a242-8116956effe6-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.609528 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.609481 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.610204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.610171 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.611023 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.610959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.616465 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.616420 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xkm\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-kube-api-access-99xkm\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.617298 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.617273 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-cqb7d\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.670403 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.670370 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:12.837536 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.837494 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" event={"ID":"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e","Type":"ContainerStarted","Data":"f751c129110fe609e38d48463ab6ce127fccda4450dc38b40c8208e863502358"} Sep 10 02:39:12.960792 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:12.960720 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d"] Sep 10 02:39:12.964433 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:39:12.964400 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dff17f8_501c_4dea_a242_8116956effe6.slice/crio-01ae7308ab48ddf179beed632b71d1820c5c6297065a25d20035dd323926e7ac WatchSource:0}: Error finding container 01ae7308ab48ddf179beed632b71d1820c5c6297065a25d20035dd323926e7ac: Status 404 returned error can't find the container with id 01ae7308ab48ddf179beed632b71d1820c5c6297065a25d20035dd323926e7ac Sep 10 02:39:13.397413 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:13.397333 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:13.843807 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:13.843773 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" event={"ID":"2dff17f8-501c-4dea-a242-8116956effe6","Type":"ContainerStarted","Data":"01ae7308ab48ddf179beed632b71d1820c5c6297065a25d20035dd323926e7ac"} Sep 10 02:39:14.810245 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:14.810221 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 02:39:15.857658 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:15.857618 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" event={"ID":"97bd02c0-6aeb-49c0-ba0c-3f4521345b6e","Type":"ContainerStarted","Data":"b89be97d17c8248f7d80ff55d6305c5a5c2168fa41e37aaf116bf35ef45622a4"} Sep 10 02:39:15.876592 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:15.876540 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74d6b6cf86-z6zzd" podStartSLOduration=3.051830959 podStartE2EDuration="5.876523399s" totalImagesPullingTime="2.82469244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:39:15.874428611 +0000 UTC m=+365.140525005" watchObservedRunningTime="2026-09-10 02:39:15.876523399 +0000 UTC m=+365.142619828" Sep 10 02:39:16.537543 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:16.537503 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:39:16.542609 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:16.537951 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:39:16.864332 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:16.864237 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" event={"ID":"2dff17f8-501c-4dea-a242-8116956effe6","Type":"ContainerStarted","Data":"ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af"} Sep 10 02:39:16.864693 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:16.864504 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:16.887519 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:16.887466 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" podStartSLOduration=1.317341169 podStartE2EDuration="4.887451524s" totalImagesPullingTime="3.570110355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:39:16.886962502 +0000 UTC m=+366.153058894" watchObservedRunningTime="2026-09-10 02:39:16.887451524 +0000 UTC m=+366.153547914" Sep 10 02:39:17.875660 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:17.875626 2570 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-cqb7d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 02:39:17.876049 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:17.875698 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" podUID="2dff17f8-501c-4dea-a242-8116956effe6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 02:39:18.018012 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:18.017972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:18.121141 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:18.121101 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:18.426344 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:18.426307 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:18.542415 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:18.542382 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:18.642115 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:18.642071 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:19.662997 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:19.662959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:19.762396 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:19.762357 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:20.202057 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:20.202020 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:20.428890 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:20.428852 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:20.613382 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:20.613292 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:20.875318 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:20.875235 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:29.908565 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:29.908526 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:30.047574 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:30.047539 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:31.876929 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.876893 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:31.941292 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.941257 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pztgq"] Sep 10 02:39:31.945217 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.945199 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:31.948726 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.948704 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8qlnd\"" Sep 10 02:39:31.949232 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.949213 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 02:39:31.961933 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.961905 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th"] Sep 10 02:39:31.965373 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.965348 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pztgq"] Sep 10 02:39:31.965488 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.965467 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:31.968053 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.968033 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-llxvx\"" Sep 10 02:39:31.968979 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.968961 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 02:39:31.983967 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.983940 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th"] Sep 10 02:39:31.989082 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:31.988995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:32.077315 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.077274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a94084da-440c-4184-9d87-6447ff838e10-cert\") pod \"kserve-controller-manager-d6f4b65b8-pztgq\" (UID: \"a94084da-440c-4184-9d87-6447ff838e10\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.077481 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.077328 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stsc7\" (UniqueName: \"kubernetes.io/projected/a94084da-440c-4184-9d87-6447ff838e10-kube-api-access-stsc7\") pod \"kserve-controller-manager-d6f4b65b8-pztgq\" (UID: \"a94084da-440c-4184-9d87-6447ff838e10\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.077481 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.077430 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzmc\" (UniqueName: \"kubernetes.io/projected/49019557-6a1c-4039-ac2f-78844c56798d-kube-api-access-ghzmc\") pod \"llmisvc-controller-manager-777c6bd7d5-gc4th\" (UID: \"49019557-6a1c-4039-ac2f-78844c56798d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:32.077607 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.077479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49019557-6a1c-4039-ac2f-78844c56798d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gc4th\" (UID: \"49019557-6a1c-4039-ac2f-78844c56798d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:32.078624 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.078591 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:32.178764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.178726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzmc\" (UniqueName: \"kubernetes.io/projected/49019557-6a1c-4039-ac2f-78844c56798d-kube-api-access-ghzmc\") pod \"llmisvc-controller-manager-777c6bd7d5-gc4th\" (UID: \"49019557-6a1c-4039-ac2f-78844c56798d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:32.178963 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.178789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49019557-6a1c-4039-ac2f-78844c56798d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gc4th\" (UID: \"49019557-6a1c-4039-ac2f-78844c56798d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:32.178963 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.178859 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a94084da-440c-4184-9d87-6447ff838e10-cert\") pod \"kserve-controller-manager-d6f4b65b8-pztgq\" (UID: \"a94084da-440c-4184-9d87-6447ff838e10\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.178963 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.178892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stsc7\" (UniqueName: \"kubernetes.io/projected/a94084da-440c-4184-9d87-6447ff838e10-kube-api-access-stsc7\") pod \"kserve-controller-manager-d6f4b65b8-pztgq\" (UID: \"a94084da-440c-4184-9d87-6447ff838e10\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.179131 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:39:32.179008 2570 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 10 02:39:32.179131 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:39:32.179082 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94084da-440c-4184-9d87-6447ff838e10-cert podName:a94084da-440c-4184-9d87-6447ff838e10 nodeName:}" failed. No retries permitted until 2026-09-10 02:39:32.679066209 +0000 UTC m=+381.945162591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a94084da-440c-4184-9d87-6447ff838e10-cert") pod "kserve-controller-manager-d6f4b65b8-pztgq" (UID: "a94084da-440c-4184-9d87-6447ff838e10") : secret "kserve-webhook-server-cert" not found Sep 10 02:39:32.181222 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.181205 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49019557-6a1c-4039-ac2f-78844c56798d-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gc4th\" (UID: \"49019557-6a1c-4039-ac2f-78844c56798d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:32.208360 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.208334 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stsc7\" (UniqueName: \"kubernetes.io/projected/a94084da-440c-4184-9d87-6447ff838e10-kube-api-access-stsc7\") pod \"kserve-controller-manager-d6f4b65b8-pztgq\" (UID: \"a94084da-440c-4184-9d87-6447ff838e10\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.208470 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.208434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzmc\" (UniqueName: \"kubernetes.io/projected/49019557-6a1c-4039-ac2f-78844c56798d-kube-api-access-ghzmc\") pod \"llmisvc-controller-manager-777c6bd7d5-gc4th\" (UID: \"49019557-6a1c-4039-ac2f-78844c56798d\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:32.258165 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.258128 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:32.277037 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.277004 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:32.417218 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.417188 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:32.543510 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.543399 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:32.553336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.553313 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th"] Sep 10 02:39:32.555181 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:39:32.555157 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49019557_6a1c_4039_ac2f_78844c56798d.slice/crio-b2957520b5f6cfee0f0aca6135f2172b6a71859df65ad187a9d0a7e2a498ebe9 WatchSource:0}: Error finding container b2957520b5f6cfee0f0aca6135f2172b6a71859df65ad187a9d0a7e2a498ebe9: Status 404 returned error can't find the container with id b2957520b5f6cfee0f0aca6135f2172b6a71859df65ad187a9d0a7e2a498ebe9 Sep 10 02:39:32.682466 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.682435 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a94084da-440c-4184-9d87-6447ff838e10-cert\") pod \"kserve-controller-manager-d6f4b65b8-pztgq\" (UID: \"a94084da-440c-4184-9d87-6447ff838e10\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.684753 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.684729 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a94084da-440c-4184-9d87-6447ff838e10-cert\") pod \"kserve-controller-manager-d6f4b65b8-pztgq\" (UID: \"a94084da-440c-4184-9d87-6447ff838e10\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.857026 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.856953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:32.937184 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:32.937152 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" event={"ID":"49019557-6a1c-4039-ac2f-78844c56798d","Type":"ContainerStarted","Data":"b2957520b5f6cfee0f0aca6135f2172b6a71859df65ad187a9d0a7e2a498ebe9"} Sep 10 02:39:33.144941 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.144916 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pztgq"] Sep 10 02:39:33.146573 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:39:33.146549 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94084da_440c_4184_9d87_6447ff838e10.slice/crio-e14365f06a59a6e5983ee82635a5d3cb27c3f8d55dceee7dd27b5aa334b49b2f WatchSource:0}: Error finding container e14365f06a59a6e5983ee82635a5d3cb27c3f8d55dceee7dd27b5aa334b49b2f: Status 404 returned error can't find the container with id e14365f06a59a6e5983ee82635a5d3cb27c3f8d55dceee7dd27b5aa334b49b2f Sep 10 02:39:33.621167 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.621130 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d"] Sep 10 02:39:33.621385 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.621360 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" podUID="2dff17f8-501c-4dea-a242-8116956effe6" containerName="discovery" containerID="cri-o://ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af" gracePeriod=30 Sep 10 02:39:33.914018 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.913986 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:33.943958 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.943908 2570 generic.go:356] "Generic (PLEG): container finished" podID="2dff17f8-501c-4dea-a242-8116956effe6" containerID="ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af" exitCode=0 Sep 10 02:39:33.944430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.944062 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" Sep 10 02:39:33.944430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.944201 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" event={"ID":"2dff17f8-501c-4dea-a242-8116956effe6","Type":"ContainerDied","Data":"ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af"} Sep 10 02:39:33.944430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.944243 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d" event={"ID":"2dff17f8-501c-4dea-a242-8116956effe6","Type":"ContainerDied","Data":"01ae7308ab48ddf179beed632b71d1820c5c6297065a25d20035dd323926e7ac"} Sep 10 02:39:33.944430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.944269 2570 scope.go:117] "RemoveContainer" containerID="ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af" Sep 10 02:39:33.945850 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.945811 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" event={"ID":"a94084da-440c-4184-9d87-6447ff838e10","Type":"ContainerStarted","Data":"e14365f06a59a6e5983ee82635a5d3cb27c3f8d55dceee7dd27b5aa334b49b2f"} Sep 10 02:39:33.956414 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.956388 2570 scope.go:117] "RemoveContainer" containerID="ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af" Sep 10 02:39:33.957204 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:39:33.956905 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af\": container with ID starting with ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af not found: ID does not exist" containerID="ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af" Sep 10 02:39:33.957204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.956946 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af"} err="failed to get container status \"ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af\": rpc error: code = NotFound desc = could not find container \"ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af\": container with ID starting with ee7cdf12e4f2e8b1d122581ae2775f0773c6f1ca1fc656adc5e4c1f7cece26af not found: ID does not exist" Sep 10 02:39:33.991727 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.991683 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99xkm\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-kube-api-access-99xkm\") pod \"2dff17f8-501c-4dea-a242-8116956effe6\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " Sep 10 02:39:33.991902 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.991741 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-dns-cert\") pod \"2dff17f8-501c-4dea-a242-8116956effe6\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " Sep 10 02:39:33.991902 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.991778 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-ca-configmap\") pod \"2dff17f8-501c-4dea-a242-8116956effe6\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " Sep 10 02:39:33.991902 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.991811 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-kubeconfig\") pod \"2dff17f8-501c-4dea-a242-8116956effe6\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " Sep 10 02:39:33.991902 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.991892 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dff17f8-501c-4dea-a242-8116956effe6-local-certs\") pod \"2dff17f8-501c-4dea-a242-8116956effe6\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " Sep 10 02:39:33.992137 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.991920 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-cacerts\") pod \"2dff17f8-501c-4dea-a242-8116956effe6\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " Sep 10 02:39:33.992137 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.991988 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-istio-token\") pod \"2dff17f8-501c-4dea-a242-8116956effe6\" (UID: \"2dff17f8-501c-4dea-a242-8116956effe6\") " Sep 10 02:39:33.992804 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.992772 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2dff17f8-501c-4dea-a242-8116956effe6" (UID: "2dff17f8-501c-4dea-a242-8116956effe6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:39:33.995117 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.995082 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-cacerts" (OuterVolumeSpecName: "cacerts") pod "2dff17f8-501c-4dea-a242-8116956effe6" (UID: "2dff17f8-501c-4dea-a242-8116956effe6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:39:33.995625 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.995583 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dff17f8-501c-4dea-a242-8116956effe6-local-certs" (OuterVolumeSpecName: "local-certs") pod "2dff17f8-501c-4dea-a242-8116956effe6" (UID: "2dff17f8-501c-4dea-a242-8116956effe6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 02:39:33.995625 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.995596 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2dff17f8-501c-4dea-a242-8116956effe6" (UID: "2dff17f8-501c-4dea-a242-8116956effe6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:39:33.996226 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.996200 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-kube-api-access-99xkm" (OuterVolumeSpecName: "kube-api-access-99xkm") pod "2dff17f8-501c-4dea-a242-8116956effe6" (UID: "2dff17f8-501c-4dea-a242-8116956effe6"). InnerVolumeSpecName "kube-api-access-99xkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:39:33.996360 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.996312 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2dff17f8-501c-4dea-a242-8116956effe6" (UID: "2dff17f8-501c-4dea-a242-8116956effe6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 02:39:33.997128 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:33.997095 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dff17f8-501c-4dea-a242-8116956effe6" (UID: "2dff17f8-501c-4dea-a242-8116956effe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:39:34.093071 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.093030 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xkm\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-kube-api-access-99xkm\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:39:34.093071 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.093068 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-dns-cert\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:39:34.093071 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.093078 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2dff17f8-501c-4dea-a242-8116956effe6-istio-csr-ca-configmap\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:39:34.093282 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.093087 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-istio-kubeconfig\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:39:34.093282 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.093096 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dff17f8-501c-4dea-a242-8116956effe6-local-certs\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:39:34.093282 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.093103 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2dff17f8-501c-4dea-a242-8116956effe6-cacerts\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:39:34.093282 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.093111 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dff17f8-501c-4dea-a242-8116956effe6-istio-token\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:39:34.271967 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.271926 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d"] Sep 10 02:39:34.278310 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:34.278274 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-cqb7d"] Sep 10 02:39:35.228087 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:35.228047 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dff17f8-501c-4dea-a242-8116956effe6" path="/var/lib/kubelet/pods/2dff17f8-501c-4dea-a242-8116956effe6/volumes" Sep 10 02:39:35.958602 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:35.958567 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" event={"ID":"49019557-6a1c-4039-ac2f-78844c56798d","Type":"ContainerStarted","Data":"955c7c54f213eb88767bc94c1cbe899e7149c741399cd7337c41a8c1a7848333"} Sep 10 02:39:35.958779 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:35.958688 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:39:35.960326 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:35.960297 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" event={"ID":"a94084da-440c-4184-9d87-6447ff838e10","Type":"ContainerStarted","Data":"efd9f897760e85cbf96f8726c18f3914da12ea90b6ae374ddfc0e7824199b5a6"} Sep 10 02:39:35.960462 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:35.960451 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:39:35.993054 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:35.992957 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" podStartSLOduration=1.809972287 podStartE2EDuration="4.992941094s" totalImagesPullingTime="3.182968807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:39:35.991515249 +0000 UTC m=+385.257611640" watchObservedRunningTime="2026-09-10 02:39:35.992941094 +0000 UTC m=+385.259037488" Sep 10 02:39:36.027159 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:36.027106 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" podStartSLOduration=2.434312912 podStartE2EDuration="5.027090437s" totalImagesPullingTime="2.592777525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:39:36.025225822 +0000 UTC m=+385.291322213" watchObservedRunningTime="2026-09-10 02:39:36.027090437 +0000 UTC m=+385.293186859" Sep 10 02:39:38.955058 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:38.955019 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:39:39.021730 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:39:39.021695 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:06.967666 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:06.967635 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gc4th" Sep 10 02:40:06.970784 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:06.970758 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pztgq" Sep 10 02:40:09.638961 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:09.638919 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:09.703773 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:09.703737 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:20.868749 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:20.868716 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:21.046644 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:21.046598 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:21.131881 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:21.131771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:21.200046 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:21.200006 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:22.753796 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:22.753753 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:33.817074 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:33.816990 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:35.883985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.883948 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv"] Sep 10 02:40:35.884380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.884248 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dff17f8-501c-4dea-a242-8116956effe6" containerName="discovery" Sep 10 02:40:35.884380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.884259 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dff17f8-501c-4dea-a242-8116956effe6" containerName="discovery" Sep 10 02:40:35.884380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.884329 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dff17f8-501c-4dea-a242-8116956effe6" containerName="discovery" Sep 10 02:40:35.887067 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.887049 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:40:35.889997 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.889969 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 02:40:35.889997 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.889991 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 02:40:35.890908 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.890887 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-b5k5b\"" Sep 10 02:40:35.898304 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.898285 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2dr\" (UniqueName: \"kubernetes.io/projected/1b95efe2-7b14-4f48-aa73-bee1a01f560f-kube-api-access-tw2dr\") pod \"limitador-operator-controller-manager-74495c69f8-xdctv\" (UID: \"1b95efe2-7b14-4f48-aa73-bee1a01f560f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:40:35.901780 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.901758 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv"] Sep 10 02:40:35.999107 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:35.999069 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2dr\" (UniqueName: \"kubernetes.io/projected/1b95efe2-7b14-4f48-aa73-bee1a01f560f-kube-api-access-tw2dr\") pod \"limitador-operator-controller-manager-74495c69f8-xdctv\" (UID: \"1b95efe2-7b14-4f48-aa73-bee1a01f560f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:40:36.011301 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:36.011270 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2dr\" (UniqueName: \"kubernetes.io/projected/1b95efe2-7b14-4f48-aa73-bee1a01f560f-kube-api-access-tw2dr\") pod \"limitador-operator-controller-manager-74495c69f8-xdctv\" (UID: \"1b95efe2-7b14-4f48-aa73-bee1a01f560f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:40:36.197375 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:36.197344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:40:36.473767 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:36.473741 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv"] Sep 10 02:40:36.476334 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:40:36.476310 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b95efe2_7b14_4f48_aa73_bee1a01f560f.slice/crio-e45ae0fcb1be7bc69e5d74ebf9ce35b9d51d6a50020a991b0dca5dd73a6d7b36 WatchSource:0}: Error finding container e45ae0fcb1be7bc69e5d74ebf9ce35b9d51d6a50020a991b0dca5dd73a6d7b36: Status 404 returned error can't find the container with id e45ae0fcb1be7bc69e5d74ebf9ce35b9d51d6a50020a991b0dca5dd73a6d7b36 Sep 10 02:40:36.688450 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:36.688413 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:37.220029 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:37.219996 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" event={"ID":"1b95efe2-7b14-4f48-aa73-bee1a01f560f","Type":"ContainerStarted","Data":"e45ae0fcb1be7bc69e5d74ebf9ce35b9d51d6a50020a991b0dca5dd73a6d7b36"} Sep 10 02:40:37.988763 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:37.988731 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g"] Sep 10 02:40:37.992002 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:37.991981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" Sep 10 02:40:37.994712 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:37.994690 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dd2hl\"" Sep 10 02:40:37.995047 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:37.995025 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 02:40:38.012177 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.012139 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgq8\" (UniqueName: \"kubernetes.io/projected/4897925a-0ae0-4a75-9c4e-34857cbb1e99-kube-api-access-kwgq8\") pod \"dns-operator-controller-manager-7b84cd7c9c-77t4g\" (UID: \"4897925a-0ae0-4a75-9c4e-34857cbb1e99\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" Sep 10 02:40:38.014885 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.014858 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g"] Sep 10 02:40:38.112933 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.112894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgq8\" (UniqueName: \"kubernetes.io/projected/4897925a-0ae0-4a75-9c4e-34857cbb1e99-kube-api-access-kwgq8\") pod \"dns-operator-controller-manager-7b84cd7c9c-77t4g\" (UID: \"4897925a-0ae0-4a75-9c4e-34857cbb1e99\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" Sep 10 02:40:38.127541 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.127508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgq8\" (UniqueName: \"kubernetes.io/projected/4897925a-0ae0-4a75-9c4e-34857cbb1e99-kube-api-access-kwgq8\") pod \"dns-operator-controller-manager-7b84cd7c9c-77t4g\" (UID: \"4897925a-0ae0-4a75-9c4e-34857cbb1e99\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" Sep 10 02:40:38.304934 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.304859 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" Sep 10 02:40:38.457135 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.457096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:38.658785 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.658705 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:38.825252 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:38.825157 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:39.229687 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:39.229656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" event={"ID":"1b95efe2-7b14-4f48-aa73-bee1a01f560f","Type":"ContainerStarted","Data":"e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c"} Sep 10 02:40:39.229912 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:39.229717 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:40:39.256464 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:39.256348 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" podStartSLOduration=1.655143318 podStartE2EDuration="4.256330815s" totalImagesPullingTime="2.601187497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:40:39.255702349 +0000 UTC m=+448.521798741" watchObservedRunningTime="2026-09-10 02:40:39.256330815 +0000 UTC m=+448.522427207" Sep 10 02:40:39.280416 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:39.280392 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g"] Sep 10 02:40:39.282727 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:40:39.282697 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4897925a_0ae0_4a75_9c4e_34857cbb1e99.slice/crio-7b4e7cfffb6d55be8f01ebb5fcd9ca20f615dc8cc4af9f492bfe37059655e05e WatchSource:0}: Error finding container 7b4e7cfffb6d55be8f01ebb5fcd9ca20f615dc8cc4af9f492bfe37059655e05e: Status 404 returned error can't find the container with id 7b4e7cfffb6d55be8f01ebb5fcd9ca20f615dc8cc4af9f492bfe37059655e05e Sep 10 02:40:39.412185 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:39.412133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:39.938568 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:39.938531 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:40.236516 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:40.236427 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" event={"ID":"4897925a-0ae0-4a75-9c4e-34857cbb1e99","Type":"ContainerStarted","Data":"7b4e7cfffb6d55be8f01ebb5fcd9ca20f615dc8cc4af9f492bfe37059655e05e"} Sep 10 02:40:40.865190 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:40.865152 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:40.967457 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:40.967429 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:41.242256 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:41.242214 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" event={"ID":"4897925a-0ae0-4a75-9c4e-34857cbb1e99","Type":"ContainerStarted","Data":"33ca21795e3726801b8f0f286ab08e9f687e0ccf654099da420eed965da2ef3e"} Sep 10 02:40:41.242450 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:41.242410 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" Sep 10 02:40:42.560057 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.559997 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" podStartSLOduration=3.9220919199999997 podStartE2EDuration="5.559977587s" totalImagesPullingTime="1.637885667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:40:41.26699249 +0000 UTC m=+450.533088880" watchObservedRunningTime="2026-09-10 02:40:42.559977587 +0000 UTC m=+451.826073956" Sep 10 02:40:42.561772 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.561746 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45"] Sep 10 02:40:42.565397 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.565374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:42.568482 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.568458 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fbrpl\"" Sep 10 02:40:42.579767 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.579735 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45"] Sep 10 02:40:42.632129 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.632096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:42.646164 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.646135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:42.646273 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.646170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vhtb\" (UniqueName: \"kubernetes.io/projected/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-kube-api-access-9vhtb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:42.747555 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.747524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:42.747555 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.747557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vhtb\" (UniqueName: \"kubernetes.io/projected/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-kube-api-access-9vhtb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:42.747951 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.747930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:42.767192 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.767166 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vhtb\" (UniqueName: \"kubernetes.io/projected/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-kube-api-access-9vhtb\") pod \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:42.880167 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:42.880083 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:43.018068 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:43.018030 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:43.168062 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:43.167981 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45"] Sep 10 02:40:43.172612 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:40:43.172583 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78568f6_5ead_4a6f_83ff_5bbd29d5fdad.slice/crio-c5f78d109360d473f01edaaacb40ee22daa1135486a90a059e66af346688c7f4 WatchSource:0}: Error finding container c5f78d109360d473f01edaaacb40ee22daa1135486a90a059e66af346688c7f4: Status 404 returned error can't find the container with id c5f78d109360d473f01edaaacb40ee22daa1135486a90a059e66af346688c7f4 Sep 10 02:40:43.253301 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:43.253268 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" event={"ID":"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad","Type":"ContainerStarted","Data":"c5f78d109360d473f01edaaacb40ee22daa1135486a90a059e66af346688c7f4"} Sep 10 02:40:44.914113 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:44.914070 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:45.023874 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.023807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:45.325460 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.325425 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-v69vk"] Sep 10 02:40:45.329104 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.329079 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" Sep 10 02:40:45.331878 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.331857 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8chb7\"" Sep 10 02:40:45.340145 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.340102 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-v69vk"] Sep 10 02:40:45.367022 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.366991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cmg\" (UniqueName: \"kubernetes.io/projected/05b528f6-094d-4a0f-a11f-7c15cb8c0a58-kube-api-access-67cmg\") pod \"authorino-operator-76b478bd78-v69vk\" (UID: \"05b528f6-094d-4a0f-a11f-7c15cb8c0a58\") " pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" Sep 10 02:40:45.468161 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.468119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cmg\" (UniqueName: \"kubernetes.io/projected/05b528f6-094d-4a0f-a11f-7c15cb8c0a58-kube-api-access-67cmg\") pod \"authorino-operator-76b478bd78-v69vk\" (UID: \"05b528f6-094d-4a0f-a11f-7c15cb8c0a58\") " pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" Sep 10 02:40:45.479158 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.479128 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cmg\" (UniqueName: \"kubernetes.io/projected/05b528f6-094d-4a0f-a11f-7c15cb8c0a58-kube-api-access-67cmg\") pod \"authorino-operator-76b478bd78-v69vk\" (UID: \"05b528f6-094d-4a0f-a11f-7c15cb8c0a58\") " pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" Sep 10 02:40:45.642471 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.642387 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" Sep 10 02:40:45.927501 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:45.927462 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-v69vk"] Sep 10 02:40:45.930641 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:40:45.930613 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b528f6_094d_4a0f_a11f_7c15cb8c0a58.slice/crio-1004589410e82d0c2b2951e08bf6a1e94c6a5adcc311cdfec0b307dcdff0e825 WatchSource:0}: Error finding container 1004589410e82d0c2b2951e08bf6a1e94c6a5adcc311cdfec0b307dcdff0e825: Status 404 returned error can't find the container with id 1004589410e82d0c2b2951e08bf6a1e94c6a5adcc311cdfec0b307dcdff0e825 Sep 10 02:40:46.270083 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:46.269997 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" event={"ID":"05b528f6-094d-4a0f-a11f-7c15cb8c0a58","Type":"ContainerStarted","Data":"1004589410e82d0c2b2951e08bf6a1e94c6a5adcc311cdfec0b307dcdff0e825"} Sep 10 02:40:46.435321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:46.435281 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:46.539676 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:46.539586 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:48.284563 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:48.284524 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" event={"ID":"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad","Type":"ContainerStarted","Data":"b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b"} Sep 10 02:40:48.285001 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:48.284640 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:40:48.311404 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:48.311354 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" podStartSLOduration=2.041584934 podStartE2EDuration="6.311338083s" totalImagesPullingTime="4.269753149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:40:48.308871068 +0000 UTC m=+457.574967459" watchObservedRunningTime="2026-09-10 02:40:48.311338083 +0000 UTC m=+457.577434473" Sep 10 02:40:48.417175 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:48.417129 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:48.541564 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:48.541472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:49.290123 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:49.290076 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" event={"ID":"05b528f6-094d-4a0f-a11f-7c15cb8c0a58","Type":"ContainerStarted","Data":"2133210a544220cddd3c401dcbf91f5afe528753ac78213c5eddf908367222ca"} Sep 10 02:40:49.290500 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:49.290387 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" Sep 10 02:40:49.310454 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:49.310406 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" podStartSLOduration=1.814600143 podStartE2EDuration="4.310393974s" totalImagesPullingTime="2.495793831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:40:49.308102151 +0000 UTC m=+458.574198541" watchObservedRunningTime="2026-09-10 02:40:49.310393974 +0000 UTC m=+458.576490365" Sep 10 02:40:50.159974 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:50.159935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:50.239191 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:50.239153 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:40:50.287772 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:50.287737 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:52.250985 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:52.250955 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-77t4g" Sep 10 02:40:54.201188 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:54.201154 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:54.301145 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:54.301106 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:54.400758 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:54.400722 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:40:59.292300 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:40:59.292269 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:00.297591 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:00.297557 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-v69vk" Sep 10 02:41:01.715074 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.715039 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45"] Sep 10 02:41:01.715471 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.715204 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" containerName="manager" containerID="cri-o://b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b" gracePeriod=2 Sep 10 02:41:01.730631 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.730594 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45"] Sep 10 02:41:01.730947 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.730908 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-75l45\": pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:01.741042 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.741008 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv"] Sep 10 02:41:01.741492 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.741250 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" containerName="manager" containerID="cri-o://e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c" gracePeriod=2 Sep 10 02:41:01.743961 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.743809 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:01.744965 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.744942 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn"] Sep 10 02:41:01.745381 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.745349 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" containerName="manager" Sep 10 02:41:01.745381 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.745370 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" containerName="manager" Sep 10 02:41:01.745530 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.745479 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" containerName="manager" Sep 10 02:41:01.748474 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.748442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:01.760377 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.760348 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv"] Sep 10 02:41:01.769089 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.769063 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn"] Sep 10 02:41:01.769944 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.769902 2570 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1b95efe2-7b14-4f48-aa73-bee1a01f560f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-xdctv\": pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:01.772261 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.772230 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:01.778191 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.778164 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2"] Sep 10 02:41:01.778520 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.778506 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" containerName="manager" Sep 10 02:41:01.778571 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.778522 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" containerName="manager" Sep 10 02:41:01.778611 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.778601 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" containerName="manager" Sep 10 02:41:01.781732 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.781710 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" Sep 10 02:41:01.801094 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.801069 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42gf\" (UniqueName: \"kubernetes.io/projected/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-kube-api-access-q42gf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6n8jn\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:01.801198 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.801171 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6n8jn\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:01.815277 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.815248 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2"] Sep 10 02:41:01.818596 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.818565 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:01.821972 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.821941 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:01.846251 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.846214 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:01.901678 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.901643 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6n8jn\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:01.901678 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.901684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg725\" (UniqueName: \"kubernetes.io/projected/6c970b0b-1057-46ca-9c02-7819728eb71e-kube-api-access-cg725\") pod \"limitador-operator-controller-manager-74495c69f8-q2pp2\" (UID: \"6c970b0b-1057-46ca-9c02-7819728eb71e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" Sep 10 02:41:01.901889 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.901741 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42gf\" (UniqueName: \"kubernetes.io/projected/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-kube-api-access-q42gf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6n8jn\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:01.902222 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.902199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6n8jn\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:01.918152 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.918061 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42gf\" (UniqueName: \"kubernetes.io/projected/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-kube-api-access-q42gf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-6n8jn\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:01.978453 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.978432 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:01.980514 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:01.980496 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:01.980694 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.980675 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:01.982610 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.982586 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:01.985246 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.985225 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:01.987957 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:01.987934 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:02.002287 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.002261 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-extensions-socket-volume\") pod \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " Sep 10 02:41:02.002399 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.002321 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vhtb\" (UniqueName: \"kubernetes.io/projected/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-kube-api-access-9vhtb\") pod \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\" (UID: \"a78568f6-5ead-4a6f-83ff-5bbd29d5fdad\") " Sep 10 02:41:02.002399 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.002385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg725\" (UniqueName: \"kubernetes.io/projected/6c970b0b-1057-46ca-9c02-7819728eb71e-kube-api-access-cg725\") pod \"limitador-operator-controller-manager-74495c69f8-q2pp2\" (UID: \"6c970b0b-1057-46ca-9c02-7819728eb71e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" Sep 10 02:41:02.002767 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.002743 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" (UID: "a78568f6-5ead-4a6f-83ff-5bbd29d5fdad"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 02:41:02.004526 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.004507 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-kube-api-access-9vhtb" (OuterVolumeSpecName: "kube-api-access-9vhtb") pod "a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" (UID: "a78568f6-5ead-4a6f-83ff-5bbd29d5fdad"). InnerVolumeSpecName "kube-api-access-9vhtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:41:02.016591 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.016566 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg725\" (UniqueName: \"kubernetes.io/projected/6c970b0b-1057-46ca-9c02-7819728eb71e-kube-api-access-cg725\") pod \"limitador-operator-controller-manager-74495c69f8-q2pp2\" (UID: \"6c970b0b-1057-46ca-9c02-7819728eb71e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" Sep 10 02:41:02.103572 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.103536 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw2dr\" (UniqueName: \"kubernetes.io/projected/1b95efe2-7b14-4f48-aa73-bee1a01f560f-kube-api-access-tw2dr\") pod \"1b95efe2-7b14-4f48-aa73-bee1a01f560f\" (UID: \"1b95efe2-7b14-4f48-aa73-bee1a01f560f\") " Sep 10 02:41:02.103764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.103707 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vhtb\" (UniqueName: \"kubernetes.io/projected/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-kube-api-access-9vhtb\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:02.103764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.103718 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad-extensions-socket-volume\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:02.105628 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.105604 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b95efe2-7b14-4f48-aa73-bee1a01f560f-kube-api-access-tw2dr" (OuterVolumeSpecName: "kube-api-access-tw2dr") pod "1b95efe2-7b14-4f48-aa73-bee1a01f560f" (UID: "1b95efe2-7b14-4f48-aa73-bee1a01f560f"). InnerVolumeSpecName "kube-api-access-tw2dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:41:02.167595 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.167555 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:02.173788 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.173768 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" Sep 10 02:41:02.204528 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.204488 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw2dr\" (UniqueName: \"kubernetes.io/projected/1b95efe2-7b14-4f48-aa73-bee1a01f560f-kube-api-access-tw2dr\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:02.349040 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.349005 2570 generic.go:356] "Generic (PLEG): container finished" podID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" containerID="e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c" exitCode=0 Sep 10 02:41:02.349215 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.349061 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:02.349215 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.349107 2570 scope.go:117] "RemoveContainer" containerID="e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c" Sep 10 02:41:02.350694 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.350668 2570 generic.go:356] "Generic (PLEG): container finished" podID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" containerID="b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b" exitCode=0 Sep 10 02:41:02.350816 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.350728 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:02.352130 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.352097 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:02.354149 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.354120 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:02.356420 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.356378 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:02.358463 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.358447 2570 scope.go:117] "RemoveContainer" containerID="e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c" Sep 10 02:41:02.358563 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.358534 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:02.358772 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.358753 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c\": container with ID starting with e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c not found: ID does not exist" containerID="e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c" Sep 10 02:41:02.358859 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.358781 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c"} err="failed to get container status \"e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c\": rpc error: code = NotFound desc = could not find container \"e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c\": container with ID starting with e23185454b4a26a241acda4d3ec567a2a28a6117700522ec881f6d5a8ff0a35c not found: ID does not exist" Sep 10 02:41:02.358859 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.358798 2570 scope.go:117] "RemoveContainer" containerID="b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b" Sep 10 02:41:02.360950 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.360929 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:02.362967 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.362945 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:02.365210 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.365186 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-xdctv\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-xdctv" Sep 10 02:41:02.366783 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.366766 2570 scope.go:117] "RemoveContainer" containerID="b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b" Sep 10 02:41:02.367044 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.367027 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b\": container with ID starting with b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b not found: ID does not exist" containerID="b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b" Sep 10 02:41:02.367095 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.367051 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b"} err="failed to get container status \"b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b\": rpc error: code = NotFound desc = could not find container \"b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b\": container with ID starting with b10868866a4b189627729552da9cdfb003bbec693050a2e9d40fbe1e0b4ad73b not found: ID does not exist" Sep 10 02:41:02.367705 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:02.367688 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-75l45\" is forbidden: User \"system:node:ip-10-0-143-228.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-228.ec2.internal' and this object" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-75l45" Sep 10 02:41:02.489216 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.489186 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2"] Sep 10 02:41:02.490925 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:41:02.490897 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c970b0b_1057_46ca_9c02_7819728eb71e.slice/crio-e72f994b3272b2638431e15aa63c1aaabeae044510ae1a5ecc70397b63efa0ab WatchSource:0}: Error finding container e72f994b3272b2638431e15aa63c1aaabeae044510ae1a5ecc70397b63efa0ab: Status 404 returned error can't find the container with id e72f994b3272b2638431e15aa63c1aaabeae044510ae1a5ecc70397b63efa0ab Sep 10 02:41:02.495125 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:02.495104 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn"] Sep 10 02:41:02.496929 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:41:02.496906 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274c1e90_d1c4_40bf_82bc_9f5f087fb6c1.slice/crio-78a154254d38a212a19651146a89835a33374d6e195ce10e7422146ff5685e11 WatchSource:0}: Error finding container 78a154254d38a212a19651146a89835a33374d6e195ce10e7422146ff5685e11: Status 404 returned error can't find the container with id 78a154254d38a212a19651146a89835a33374d6e195ce10e7422146ff5685e11 Sep 10 02:41:03.090144 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.090075 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:03.192546 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.192508 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:03.228544 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.228513 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b95efe2-7b14-4f48-aa73-bee1a01f560f" path="/var/lib/kubelet/pods/1b95efe2-7b14-4f48-aa73-bee1a01f560f/volumes" Sep 10 02:41:03.228888 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.228875 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a78568f6-5ead-4a6f-83ff-5bbd29d5fdad" path="/var/lib/kubelet/pods/a78568f6-5ead-4a6f-83ff-5bbd29d5fdad/volumes" Sep 10 02:41:03.358480 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.358340 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" event={"ID":"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1","Type":"ContainerStarted","Data":"885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7"} Sep 10 02:41:03.358480 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.358402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" event={"ID":"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1","Type":"ContainerStarted","Data":"78a154254d38a212a19651146a89835a33374d6e195ce10e7422146ff5685e11"} Sep 10 02:41:03.358480 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.358430 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:03.362301 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.362271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" event={"ID":"6c970b0b-1057-46ca-9c02-7819728eb71e","Type":"ContainerStarted","Data":"df756588244aec78ab263ed56d7ebb68971ee88d50013b13c28df88bb604f248"} Sep 10 02:41:03.362400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.362321 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" event={"ID":"6c970b0b-1057-46ca-9c02-7819728eb71e","Type":"ContainerStarted","Data":"e72f994b3272b2638431e15aa63c1aaabeae044510ae1a5ecc70397b63efa0ab"} Sep 10 02:41:03.362400 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.362362 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" Sep 10 02:41:03.400315 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.400190 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" podStartSLOduration=2.400170026 podStartE2EDuration="2.400170026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:41:03.39720209 +0000 UTC m=+472.663298482" watchObservedRunningTime="2026-09-10 02:41:03.400170026 +0000 UTC m=+472.666266417" Sep 10 02:41:03.459165 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:03.459109 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" podStartSLOduration=2.459091515 podStartE2EDuration="2.459091515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:41:03.457585557 +0000 UTC m=+472.723681948" watchObservedRunningTime="2026-09-10 02:41:03.459091515 +0000 UTC m=+472.725187961" Sep 10 02:41:05.680385 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:05.680343 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:05.783229 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:05.783176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:08.633865 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:08.633809 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:08.779332 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:08.779292 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:13.443080 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:13.443032 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:13.543465 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:13.543418 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:14.369048 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:14.369018 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:14.369200 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:14.369097 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-q2pp2" Sep 10 02:41:14.444243 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:14.444201 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:14.552961 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:14.552927 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:18.384447 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.384418 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn"] Sep 10 02:41:18.384835 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.384586 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" podUID="274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" containerName="manager" containerID="cri-o://885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7" gracePeriod=10 Sep 10 02:41:18.630521 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.630494 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:18.759556 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.759520 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-extensions-socket-volume\") pod \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " Sep 10 02:41:18.759725 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.759567 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q42gf\" (UniqueName: \"kubernetes.io/projected/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-kube-api-access-q42gf\") pod \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\" (UID: \"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1\") " Sep 10 02:41:18.759969 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.759947 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" (UID: "274c1e90-d1c4-40bf-82bc-9f5f087fb6c1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 02:41:18.761634 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.761611 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-kube-api-access-q42gf" (OuterVolumeSpecName: "kube-api-access-q42gf") pod "274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" (UID: "274c1e90-d1c4-40bf-82bc-9f5f087fb6c1"). InnerVolumeSpecName "kube-api-access-q42gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:41:18.860325 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.860280 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-extensions-socket-volume\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:18.860325 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:18.860319 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42gf\" (UniqueName: \"kubernetes.io/projected/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1-kube-api-access-q42gf\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:19.444136 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.444101 2570 generic.go:356] "Generic (PLEG): container finished" podID="274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" containerID="885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7" exitCode=0 Sep 10 02:41:19.444554 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.444157 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" Sep 10 02:41:19.444554 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.444191 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" event={"ID":"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1","Type":"ContainerDied","Data":"885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7"} Sep 10 02:41:19.444554 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.444282 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn" event={"ID":"274c1e90-d1c4-40bf-82bc-9f5f087fb6c1","Type":"ContainerDied","Data":"78a154254d38a212a19651146a89835a33374d6e195ce10e7422146ff5685e11"} Sep 10 02:41:19.444554 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.444310 2570 scope.go:117] "RemoveContainer" containerID="885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7" Sep 10 02:41:19.452923 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.452905 2570 scope.go:117] "RemoveContainer" containerID="885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7" Sep 10 02:41:19.453176 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:19.453161 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7\": container with ID starting with 885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7 not found: ID does not exist" containerID="885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7" Sep 10 02:41:19.453229 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.453182 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7"} err="failed to get container status \"885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7\": rpc error: code = NotFound desc = could not find container \"885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7\": container with ID starting with 885d1ad23a635facfcb507daeedc44af9216bc14d597edc055f4bf3dadaf98d7 not found: ID does not exist" Sep 10 02:41:19.474062 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.474034 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn"] Sep 10 02:41:19.479087 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:19.479066 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-6n8jn"] Sep 10 02:41:20.464605 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:20.464564 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:21.227331 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:21.227304 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" path="/var/lib/kubelet/pods/274c1e90-d1c4-40bf-82bc-9f5f087fb6c1/volumes" Sep 10 02:41:22.259784 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:22.259745 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:22.360921 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:22.360877 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:34.925922 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.925886 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz"] Sep 10 02:41:34.926281 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.926264 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" containerName="manager" Sep 10 02:41:34.926281 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.926277 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" containerName="manager" Sep 10 02:41:34.926348 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.926336 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="274c1e90-d1c4-40bf-82bc-9f5f087fb6c1" containerName="manager" Sep 10 02:41:34.930473 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.930449 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:34.933699 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.933678 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6fxs6\"" Sep 10 02:41:34.933813 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.933678 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 02:41:34.941608 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.941585 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz"] Sep 10 02:41:34.950371 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:34.950341 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:35.096271 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a55644-04be-4f11-b5a5-ff85419e3bfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096431 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096286 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096431 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096431 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096431 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096368 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096431 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096383 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096431 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmmh\" (UniqueName: \"kubernetes.io/projected/77a55644-04be-4f11-b5a5-ff85419e3bfe-kube-api-access-qqmmh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096620 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096453 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.096620 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.096468 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.143139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.143108 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:35.197162 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197089 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a55644-04be-4f11-b5a5-ff85419e3bfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197162 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197162 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197187 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197205 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197244 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmmh\" (UniqueName: \"kubernetes.io/projected/77a55644-04be-4f11-b5a5-ff85419e3bfe-kube-api-access-qqmmh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197849 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197807 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197988 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197903 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197988 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197947 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.197988 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.197968 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.198197 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.198178 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77a55644-04be-4f11-b5a5-ff85419e3bfe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.199888 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.199866 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.200012 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.199994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.205349 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.205321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmmh\" (UniqueName: \"kubernetes.io/projected/77a55644-04be-4f11-b5a5-ff85419e3bfe-kube-api-access-qqmmh\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.205432 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.205330 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77a55644-04be-4f11-b5a5-ff85419e3bfe-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-gfpvz\" (UID: \"77a55644-04be-4f11-b5a5-ff85419e3bfe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.241630 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.241602 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:35.518434 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:35.518411 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz"] Sep 10 02:41:35.520605 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:41:35.520578 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a55644_04be_4f11_b5a5_ff85419e3bfe.slice/crio-1534e74f88c5f7936532f1bd271ca676adb1be591a3a09d0f32ee4f212bd60f6 WatchSource:0}: Error finding container 1534e74f88c5f7936532f1bd271ca676adb1be591a3a09d0f32ee4f212bd60f6: Status 404 returned error can't find the container with id 1534e74f88c5f7936532f1bd271ca676adb1be591a3a09d0f32ee4f212bd60f6 Sep 10 02:41:36.525224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:36.525181 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" event={"ID":"77a55644-04be-4f11-b5a5-ff85419e3bfe","Type":"ContainerStarted","Data":"1534e74f88c5f7936532f1bd271ca676adb1be591a3a09d0f32ee4f212bd60f6"} Sep 10 02:41:38.269725 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:38.269687 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:41:38.270171 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:38.269772 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:41:38.270171 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:38.269819 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:41:38.537512 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:38.537431 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" event={"ID":"77a55644-04be-4f11-b5a5-ff85419e3bfe","Type":"ContainerStarted","Data":"a49a25034bc46f93585d9cbb5a2a7603cc4721a01cce7c3426803c720db729f3"} Sep 10 02:41:38.563076 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:38.563018 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" podStartSLOduration=1.816063896 podStartE2EDuration="4.562999566s" totalImagesPullingTime="2.74693567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:41:38.560627074 +0000 UTC m=+507.826723464" watchObservedRunningTime="2026-09-10 02:41:38.562999566 +0000 UTC m=+507.829095958" Sep 10 02:41:39.242279 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.242249 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:39.247679 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.247654 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:39.541811 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.541729 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:39.542645 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.542624 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:41:39.686334 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.686303 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xtzm2"] Sep 10 02:41:39.686509 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.686384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:39.689951 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.689934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:39.692523 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.692501 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4s7v6\"" Sep 10 02:41:39.692613 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.692501 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 02:41:39.697218 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.696964 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xtzm2"] Sep 10 02:41:39.739544 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.739518 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnl7\" (UniqueName: \"kubernetes.io/projected/45fe583a-f7b1-49ad-9035-1b4561232618-kube-api-access-ktnl7\") pod \"limitador-limitador-8588749977-xtzm2\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:39.739665 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.739552 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45fe583a-f7b1-49ad-9035-1b4561232618-config-file\") pod \"limitador-limitador-8588749977-xtzm2\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:39.781567 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.781538 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xtzm2"] Sep 10 02:41:39.791545 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.791508 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:39.840812 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.840749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnl7\" (UniqueName: \"kubernetes.io/projected/45fe583a-f7b1-49ad-9035-1b4561232618-kube-api-access-ktnl7\") pod \"limitador-limitador-8588749977-xtzm2\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:39.840812 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.840785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45fe583a-f7b1-49ad-9035-1b4561232618-config-file\") pod \"limitador-limitador-8588749977-xtzm2\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:39.841432 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.841413 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45fe583a-f7b1-49ad-9035-1b4561232618-config-file\") pod \"limitador-limitador-8588749977-xtzm2\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:39.849457 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.849435 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnl7\" (UniqueName: \"kubernetes.io/projected/45fe583a-f7b1-49ad-9035-1b4561232618-kube-api-access-ktnl7\") pod \"limitador-limitador-8588749977-xtzm2\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:39.894246 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:39.894215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:40.001243 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.001211 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:40.274999 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.274962 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xtzm2"] Sep 10 02:41:40.278066 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:41:40.278029 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45fe583a_f7b1_49ad_9035_1b4561232618.slice/crio-70f163df7cfa10bcbed726bb895dff3a00c2ded59e5d8960b355627c4b3f241b WatchSource:0}: Error finding container 70f163df7cfa10bcbed726bb895dff3a00c2ded59e5d8960b355627c4b3f241b: Status 404 returned error can't find the container with id 70f163df7cfa10bcbed726bb895dff3a00c2ded59e5d8960b355627c4b3f241b Sep 10 02:41:40.502291 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.502257 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mzdkp"] Sep 10 02:41:40.506992 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.506971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" Sep 10 02:41:40.509927 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.509901 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nfn99\"" Sep 10 02:41:40.514745 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.514721 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mzdkp"] Sep 10 02:41:40.536105 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.536043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:40.545893 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.545868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcj8t\" (UniqueName: \"kubernetes.io/projected/0eea2a77-606c-4ec7-a4cc-aaa83cab9b87-kube-api-access-lcj8t\") pod \"authorino-7bcfd8f9cd-mzdkp\" (UID: \"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" Sep 10 02:41:40.546957 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.546929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" event={"ID":"45fe583a-f7b1-49ad-9035-1b4561232618","Type":"ContainerStarted","Data":"70f163df7cfa10bcbed726bb895dff3a00c2ded59e5d8960b355627c4b3f241b"} Sep 10 02:41:40.627854 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.627784 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:40.631817 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.631798 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-6k4pl"] Sep 10 02:41:40.635090 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.635074 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6k4pl" Sep 10 02:41:40.644771 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.644751 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6k4pl"] Sep 10 02:41:40.647051 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.647025 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgx6\" (UniqueName: \"kubernetes.io/projected/eaa41136-e9ec-4902-964f-4703ae1b76e1-kube-api-access-tpgx6\") pod \"authorino-59c974bd48-6k4pl\" (UID: \"eaa41136-e9ec-4902-964f-4703ae1b76e1\") " pod="kuadrant-system/authorino-59c974bd48-6k4pl" Sep 10 02:41:40.647216 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.647190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcj8t\" (UniqueName: \"kubernetes.io/projected/0eea2a77-606c-4ec7-a4cc-aaa83cab9b87-kube-api-access-lcj8t\") pod \"authorino-7bcfd8f9cd-mzdkp\" (UID: \"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" Sep 10 02:41:40.662523 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.662499 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcj8t\" (UniqueName: \"kubernetes.io/projected/0eea2a77-606c-4ec7-a4cc-aaa83cab9b87-kube-api-access-lcj8t\") pod \"authorino-7bcfd8f9cd-mzdkp\" (UID: \"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" Sep 10 02:41:40.748271 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.748227 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgx6\" (UniqueName: \"kubernetes.io/projected/eaa41136-e9ec-4902-964f-4703ae1b76e1-kube-api-access-tpgx6\") pod \"authorino-59c974bd48-6k4pl\" (UID: \"eaa41136-e9ec-4902-964f-4703ae1b76e1\") " pod="kuadrant-system/authorino-59c974bd48-6k4pl" Sep 10 02:41:40.764729 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.764704 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgx6\" (UniqueName: \"kubernetes.io/projected/eaa41136-e9ec-4902-964f-4703ae1b76e1-kube-api-access-tpgx6\") pod \"authorino-59c974bd48-6k4pl\" (UID: \"eaa41136-e9ec-4902-964f-4703ae1b76e1\") " pod="kuadrant-system/authorino-59c974bd48-6k4pl" Sep 10 02:41:40.816587 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.816514 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" Sep 10 02:41:40.944170 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:40.944131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6k4pl" Sep 10 02:41:41.085806 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:41.085734 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mzdkp"] Sep 10 02:41:41.088796 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:41:41.088772 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eea2a77_606c_4ec7_a4cc_aaa83cab9b87.slice/crio-2e45ea0bc38233c71853e300fa7a0d72b942410c7451abdd3d1f66b9e47a0cb0 WatchSource:0}: Error finding container 2e45ea0bc38233c71853e300fa7a0d72b942410c7451abdd3d1f66b9e47a0cb0: Status 404 returned error can't find the container with id 2e45ea0bc38233c71853e300fa7a0d72b942410c7451abdd3d1f66b9e47a0cb0 Sep 10 02:41:41.211661 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:41.211636 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6k4pl"] Sep 10 02:41:41.214013 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:41:41.213984 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaa41136_e9ec_4902_964f_4703ae1b76e1.slice/crio-1c10f06f90c312b5ea0604d01917e62a3bfcc057d067efea6675832ad7e2e086 WatchSource:0}: Error finding container 1c10f06f90c312b5ea0604d01917e62a3bfcc057d067efea6675832ad7e2e086: Status 404 returned error can't find the container with id 1c10f06f90c312b5ea0604d01917e62a3bfcc057d067efea6675832ad7e2e086 Sep 10 02:41:41.555663 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:41.555622 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6k4pl" event={"ID":"eaa41136-e9ec-4902-964f-4703ae1b76e1","Type":"ContainerStarted","Data":"1c10f06f90c312b5ea0604d01917e62a3bfcc057d067efea6675832ad7e2e086"} Sep 10 02:41:41.556777 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:41.556754 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" event={"ID":"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87","Type":"ContainerStarted","Data":"2e45ea0bc38233c71853e300fa7a0d72b942410c7451abdd3d1f66b9e47a0cb0"} Sep 10 02:41:43.568512 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:43.568476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" event={"ID":"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87","Type":"ContainerStarted","Data":"c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7"} Sep 10 02:41:43.583266 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:43.583215 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" podStartSLOduration=1.637306461 podStartE2EDuration="3.583200347s" totalImagesPullingTime="1.945893886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:41:43.582773907 +0000 UTC m=+512.848870300" watchObservedRunningTime="2026-09-10 02:41:43.583200347 +0000 UTC m=+512.849296738" Sep 10 02:41:45.579775 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.579739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6k4pl" event={"ID":"eaa41136-e9ec-4902-964f-4703ae1b76e1","Type":"ContainerStarted","Data":"78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021"} Sep 10 02:41:45.594989 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.594931 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-6k4pl" podStartSLOduration=1.77560493 podStartE2EDuration="5.594912432s" totalImagesPullingTime="3.819307502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:41:45.59446532 +0000 UTC m=+514.860561711" watchObservedRunningTime="2026-09-10 02:41:45.594912432 +0000 UTC m=+514.861008824" Sep 10 02:41:45.623491 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.623452 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mzdkp"] Sep 10 02:41:45.623672 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.623647 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" podUID="0eea2a77-606c-4ec7-a4cc-aaa83cab9b87" containerName="authorino" containerID="cri-o://c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7" gracePeriod=30 Sep 10 02:41:45.879371 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.879344 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" Sep 10 02:41:45.892862 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.892815 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcj8t\" (UniqueName: \"kubernetes.io/projected/0eea2a77-606c-4ec7-a4cc-aaa83cab9b87-kube-api-access-lcj8t\") pod \"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87\" (UID: \"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87\") " Sep 10 02:41:45.894861 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.894807 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eea2a77-606c-4ec7-a4cc-aaa83cab9b87-kube-api-access-lcj8t" (OuterVolumeSpecName: "kube-api-access-lcj8t") pod "0eea2a77-606c-4ec7-a4cc-aaa83cab9b87" (UID: "0eea2a77-606c-4ec7-a4cc-aaa83cab9b87"). InnerVolumeSpecName "kube-api-access-lcj8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:41:45.994345 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:45.994306 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcj8t\" (UniqueName: \"kubernetes.io/projected/0eea2a77-606c-4ec7-a4cc-aaa83cab9b87-kube-api-access-lcj8t\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:46.404973 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.404938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:46.466446 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.466413 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:46.585727 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.585691 2570 generic.go:356] "Generic (PLEG): container finished" podID="0eea2a77-606c-4ec7-a4cc-aaa83cab9b87" containerID="c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7" exitCode=0 Sep 10 02:41:46.586209 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.585734 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" Sep 10 02:41:46.586209 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.585776 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" event={"ID":"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87","Type":"ContainerDied","Data":"c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7"} Sep 10 02:41:46.586209 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.585819 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mzdkp" event={"ID":"0eea2a77-606c-4ec7-a4cc-aaa83cab9b87","Type":"ContainerDied","Data":"2e45ea0bc38233c71853e300fa7a0d72b942410c7451abdd3d1f66b9e47a0cb0"} Sep 10 02:41:46.586209 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.585858 2570 scope.go:117] "RemoveContainer" containerID="c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7" Sep 10 02:41:46.596361 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.596335 2570 scope.go:117] "RemoveContainer" containerID="c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7" Sep 10 02:41:46.596775 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:46.596746 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7\": container with ID starting with c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7 not found: ID does not exist" containerID="c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7" Sep 10 02:41:46.596850 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.596783 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7"} err="failed to get container status \"c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7\": rpc error: code = NotFound desc = could not find container \"c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7\": container with ID starting with c5cd3ce3b614874bab371c15243e8b2011b2f5d5fc41c54cb056425886ae8dd7 not found: ID does not exist" Sep 10 02:41:46.613493 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.613468 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mzdkp"] Sep 10 02:41:46.619029 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:46.619006 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mzdkp"] Sep 10 02:41:47.227981 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:47.227948 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eea2a77-606c-4ec7-a4cc-aaa83cab9b87" path="/var/lib/kubelet/pods/0eea2a77-606c-4ec7-a4cc-aaa83cab9b87/volumes" Sep 10 02:41:47.911723 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:47.911688 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:51.612067 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:51.612030 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" event={"ID":"45fe583a-f7b1-49ad-9035-1b4561232618","Type":"ContainerStarted","Data":"d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8"} Sep 10 02:41:51.612472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:51.612091 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:51.631697 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:51.631636 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" podStartSLOduration=1.9687875080000001 podStartE2EDuration="12.6316205s" totalImagesPullingTime="10.662832992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:41:51.630370251 +0000 UTC m=+520.896466643" watchObservedRunningTime="2026-09-10 02:41:51.6316205 +0000 UTC m=+520.897716950" Sep 10 02:41:53.518906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:53.518871 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xtzm2"] Sep 10 02:41:53.621686 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:53.621646 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" podUID="45fe583a-f7b1-49ad-9035-1b4561232618" containerName="limitador" containerID="cri-o://d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8" gracePeriod=30 Sep 10 02:41:54.193864 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.193791 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:54.264944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.264905 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktnl7\" (UniqueName: \"kubernetes.io/projected/45fe583a-f7b1-49ad-9035-1b4561232618-kube-api-access-ktnl7\") pod \"45fe583a-f7b1-49ad-9035-1b4561232618\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " Sep 10 02:41:54.264944 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.264946 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45fe583a-f7b1-49ad-9035-1b4561232618-config-file\") pod \"45fe583a-f7b1-49ad-9035-1b4561232618\" (UID: \"45fe583a-f7b1-49ad-9035-1b4561232618\") " Sep 10 02:41:54.265330 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.265303 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fe583a-f7b1-49ad-9035-1b4561232618-config-file" (OuterVolumeSpecName: "config-file") pod "45fe583a-f7b1-49ad-9035-1b4561232618" (UID: "45fe583a-f7b1-49ad-9035-1b4561232618"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 02:41:54.267091 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.267071 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fe583a-f7b1-49ad-9035-1b4561232618-kube-api-access-ktnl7" (OuterVolumeSpecName: "kube-api-access-ktnl7") pod "45fe583a-f7b1-49ad-9035-1b4561232618" (UID: "45fe583a-f7b1-49ad-9035-1b4561232618"). InnerVolumeSpecName "kube-api-access-ktnl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:41:54.366262 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.366228 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktnl7\" (UniqueName: \"kubernetes.io/projected/45fe583a-f7b1-49ad-9035-1b4561232618-kube-api-access-ktnl7\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:54.366262 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.366255 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45fe583a-f7b1-49ad-9035-1b4561232618-config-file\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:41:54.633389 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.633306 2570 generic.go:356] "Generic (PLEG): container finished" podID="45fe583a-f7b1-49ad-9035-1b4561232618" containerID="d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8" exitCode=0 Sep 10 02:41:54.633389 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.633350 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" Sep 10 02:41:54.633815 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.633388 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" event={"ID":"45fe583a-f7b1-49ad-9035-1b4561232618","Type":"ContainerDied","Data":"d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8"} Sep 10 02:41:54.633815 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.633462 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xtzm2" event={"ID":"45fe583a-f7b1-49ad-9035-1b4561232618","Type":"ContainerDied","Data":"70f163df7cfa10bcbed726bb895dff3a00c2ded59e5d8960b355627c4b3f241b"} Sep 10 02:41:54.633815 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.633483 2570 scope.go:117] "RemoveContainer" containerID="d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8" Sep 10 02:41:54.642101 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.642073 2570 scope.go:117] "RemoveContainer" containerID="d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8" Sep 10 02:41:54.642495 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:41:54.642461 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8\": container with ID starting with d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8 not found: ID does not exist" containerID="d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8" Sep 10 02:41:54.642558 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.642509 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8"} err="failed to get container status \"d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8\": rpc error: code = NotFound desc = could not find container \"d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8\": container with ID starting with d546b2f099f7f96cb8236a3e91ccf015cb07fbe0f8d7f66fe55178120b8c19b8 not found: ID does not exist" Sep 10 02:41:54.654904 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.654881 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xtzm2"] Sep 10 02:41:54.658870 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:54.658846 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xtzm2"] Sep 10 02:41:55.227661 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:55.227627 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45fe583a-f7b1-49ad-9035-1b4561232618" path="/var/lib/kubelet/pods/45fe583a-f7b1-49ad-9035-1b4561232618/volumes" Sep 10 02:41:56.018383 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.018352 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:56.115927 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.115884 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:56.519214 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.519168 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc"] Sep 10 02:41:56.519529 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.519516 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fe583a-f7b1-49ad-9035-1b4561232618" containerName="limitador" Sep 10 02:41:56.519603 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.519530 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fe583a-f7b1-49ad-9035-1b4561232618" containerName="limitador" Sep 10 02:41:56.519603 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.519546 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eea2a77-606c-4ec7-a4cc-aaa83cab9b87" containerName="authorino" Sep 10 02:41:56.519603 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.519552 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eea2a77-606c-4ec7-a4cc-aaa83cab9b87" containerName="authorino" Sep 10 02:41:56.519762 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.519617 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fe583a-f7b1-49ad-9035-1b4561232618" containerName="limitador" Sep 10 02:41:56.519762 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.519625 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eea2a77-606c-4ec7-a4cc-aaa83cab9b87" containerName="authorino" Sep 10 02:41:56.524091 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.524069 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:56.526787 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.526760 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 02:41:56.526917 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.526770 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 10 02:41:56.527674 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.527658 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4bgt9\"" Sep 10 02:41:56.527729 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.527661 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 02:41:56.532616 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.532594 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc"] Sep 10 02:41:56.556157 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.556133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:41:56.587570 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.587544 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed3a2902-4b84-47c2-b44c-8c17d0e86d24-data\") pod \"postgres-69fb4594cf-gzhrc\" (UID: \"ed3a2902-4b84-47c2-b44c-8c17d0e86d24\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:56.587652 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.587579 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqkf5\" (UniqueName: \"kubernetes.io/projected/ed3a2902-4b84-47c2-b44c-8c17d0e86d24-kube-api-access-mqkf5\") pod \"postgres-69fb4594cf-gzhrc\" (UID: \"ed3a2902-4b84-47c2-b44c-8c17d0e86d24\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:56.688677 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.688639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed3a2902-4b84-47c2-b44c-8c17d0e86d24-data\") pod \"postgres-69fb4594cf-gzhrc\" (UID: \"ed3a2902-4b84-47c2-b44c-8c17d0e86d24\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:56.688677 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.688677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqkf5\" (UniqueName: \"kubernetes.io/projected/ed3a2902-4b84-47c2-b44c-8c17d0e86d24-kube-api-access-mqkf5\") pod \"postgres-69fb4594cf-gzhrc\" (UID: \"ed3a2902-4b84-47c2-b44c-8c17d0e86d24\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:56.689020 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.689001 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed3a2902-4b84-47c2-b44c-8c17d0e86d24-data\") pod \"postgres-69fb4594cf-gzhrc\" (UID: \"ed3a2902-4b84-47c2-b44c-8c17d0e86d24\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:56.697234 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.697204 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqkf5\" (UniqueName: \"kubernetes.io/projected/ed3a2902-4b84-47c2-b44c-8c17d0e86d24-kube-api-access-mqkf5\") pod \"postgres-69fb4594cf-gzhrc\" (UID: \"ed3a2902-4b84-47c2-b44c-8c17d0e86d24\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:56.835119 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:56.835040 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:41:57.103994 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:57.103967 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc"] Sep 10 02:41:57.104342 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:41:57.104277 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3a2902_4b84_47c2_b44c_8c17d0e86d24.slice/crio-f213d139767e75ba384e235e85b784c816be8e9c83103c3f57ba26b2600fdd00 WatchSource:0}: Error finding container f213d139767e75ba384e235e85b784c816be8e9c83103c3f57ba26b2600fdd00: Status 404 returned error can't find the container with id f213d139767e75ba384e235e85b784c816be8e9c83103c3f57ba26b2600fdd00 Sep 10 02:41:57.650764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:41:57.650729 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" event={"ID":"ed3a2902-4b84-47c2-b44c-8c17d0e86d24","Type":"ContainerStarted","Data":"f213d139767e75ba384e235e85b784c816be8e9c83103c3f57ba26b2600fdd00"} Sep 10 02:42:02.680266 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:02.680223 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" event={"ID":"ed3a2902-4b84-47c2-b44c-8c17d0e86d24","Type":"ContainerStarted","Data":"e10f6ac19092b69b1de33cf5538272704402bb69054269d5f34a12cbd3fbb5d2"} Sep 10 02:42:02.680738 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:02.680361 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:42:02.697593 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:02.697541 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" podStartSLOduration=1.970417651 podStartE2EDuration="6.69752231s" totalImagesPullingTime="4.727104659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:42:02.697127245 +0000 UTC m=+531.963223649" watchObservedRunningTime="2026-09-10 02:42:02.69752231 +0000 UTC m=+531.963618702" Sep 10 02:42:08.714238 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:08.714169 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gzhrc" Sep 10 02:42:09.085745 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.085644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:09.207173 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.207138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:09.336689 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.336612 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-xff2f"] Sep 10 02:42:09.343343 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.343318 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xff2f" Sep 10 02:42:09.349336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.349311 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-xff2f"] Sep 10 02:42:09.393110 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.393075 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xff2f"] Sep 10 02:42:09.393358 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:42:09.393337 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7cc6h], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-xff2f" podUID="57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39" Sep 10 02:42:09.394606 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.394582 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc6h\" (UniqueName: \"kubernetes.io/projected/57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39-kube-api-access-7cc6h\") pod \"authorino-557f876f69-xff2f\" (UID: \"57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39\") " pod="kuadrant-system/authorino-557f876f69-xff2f" Sep 10 02:42:09.495741 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.495702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc6h\" (UniqueName: \"kubernetes.io/projected/57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39-kube-api-access-7cc6h\") pod \"authorino-557f876f69-xff2f\" (UID: \"57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39\") " pod="kuadrant-system/authorino-557f876f69-xff2f" Sep 10 02:42:09.506702 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.506669 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc6h\" (UniqueName: \"kubernetes.io/projected/57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39-kube-api-access-7cc6h\") pod \"authorino-557f876f69-xff2f\" (UID: \"57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39\") " pod="kuadrant-system/authorino-557f876f69-xff2f" Sep 10 02:42:09.696953 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.696917 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c574fbcf-b97jg"] Sep 10 02:42:09.701281 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.701260 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:09.703699 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.703678 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 02:42:09.711362 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.711340 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c574fbcf-b97jg"] Sep 10 02:42:09.713608 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.713589 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xff2f" Sep 10 02:42:09.721566 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.721548 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xff2f" Sep 10 02:42:09.798728 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.798701 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cc6h\" (UniqueName: \"kubernetes.io/projected/57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39-kube-api-access-7cc6h\") pod \"57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39\" (UID: \"57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39\") " Sep 10 02:42:09.798883 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.798866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ddc0c832-e730-4741-9b9d-2d0b3e182a95-tls-cert\") pod \"authorino-56c574fbcf-b97jg\" (UID: \"ddc0c832-e730-4741-9b9d-2d0b3e182a95\") " pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:09.798949 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.798904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67m8\" (UniqueName: \"kubernetes.io/projected/ddc0c832-e730-4741-9b9d-2d0b3e182a95-kube-api-access-l67m8\") pod \"authorino-56c574fbcf-b97jg\" (UID: \"ddc0c832-e730-4741-9b9d-2d0b3e182a95\") " pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:09.800791 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.800763 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39-kube-api-access-7cc6h" (OuterVolumeSpecName: "kube-api-access-7cc6h") pod "57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39" (UID: "57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39"). InnerVolumeSpecName "kube-api-access-7cc6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:42:09.899600 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.899563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ddc0c832-e730-4741-9b9d-2d0b3e182a95-tls-cert\") pod \"authorino-56c574fbcf-b97jg\" (UID: \"ddc0c832-e730-4741-9b9d-2d0b3e182a95\") " pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:09.899755 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.899612 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67m8\" (UniqueName: \"kubernetes.io/projected/ddc0c832-e730-4741-9b9d-2d0b3e182a95-kube-api-access-l67m8\") pod \"authorino-56c574fbcf-b97jg\" (UID: \"ddc0c832-e730-4741-9b9d-2d0b3e182a95\") " pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:09.899755 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.899680 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cc6h\" (UniqueName: \"kubernetes.io/projected/57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39-kube-api-access-7cc6h\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:42:09.902085 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.902063 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ddc0c832-e730-4741-9b9d-2d0b3e182a95-tls-cert\") pod \"authorino-56c574fbcf-b97jg\" (UID: \"ddc0c832-e730-4741-9b9d-2d0b3e182a95\") " pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:09.909273 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:09.909245 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67m8\" (UniqueName: \"kubernetes.io/projected/ddc0c832-e730-4741-9b9d-2d0b3e182a95-kube-api-access-l67m8\") pod \"authorino-56c574fbcf-b97jg\" (UID: \"ddc0c832-e730-4741-9b9d-2d0b3e182a95\") " pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:10.011355 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:10.011264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c574fbcf-b97jg" Sep 10 02:42:10.284070 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:10.284042 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c574fbcf-b97jg"] Sep 10 02:42:10.286411 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:42:10.286385 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddc0c832_e730_4741_9b9d_2d0b3e182a95.slice/crio-26d68af5c6e31c4beeb4bef1823ffbac6ed232870015812ad583fd40a5af2a7e WatchSource:0}: Error finding container 26d68af5c6e31c4beeb4bef1823ffbac6ed232870015812ad583fd40a5af2a7e: Status 404 returned error can't find the container with id 26d68af5c6e31c4beeb4bef1823ffbac6ed232870015812ad583fd40a5af2a7e Sep 10 02:42:10.719598 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:10.719562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c574fbcf-b97jg" event={"ID":"ddc0c832-e730-4741-9b9d-2d0b3e182a95","Type":"ContainerStarted","Data":"26d68af5c6e31c4beeb4bef1823ffbac6ed232870015812ad583fd40a5af2a7e"} Sep 10 02:42:10.719797 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:10.719610 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xff2f" Sep 10 02:42:10.749152 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:10.749124 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xff2f"] Sep 10 02:42:10.751104 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:10.751065 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-xff2f"] Sep 10 02:42:11.229607 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:11.229582 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39" path="/var/lib/kubelet/pods/57a38d5c-9cd9-49f0-976c-b2a7bcbd1a39/volumes" Sep 10 02:42:11.726290 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:11.726246 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c574fbcf-b97jg" event={"ID":"ddc0c832-e730-4741-9b9d-2d0b3e182a95","Type":"ContainerStarted","Data":"c9988cc2822567a19290ab3f3cf87065536b488a813ae3f2c6496c165c05870a"} Sep 10 02:42:11.746721 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:11.746664 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56c574fbcf-b97jg" podStartSLOduration=2.288105576 podStartE2EDuration="2.746648677s" totalImagesPullingTime="458.543101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:42:11.745539669 +0000 UTC m=+541.011636057" watchObservedRunningTime="2026-09-10 02:42:11.746648677 +0000 UTC m=+541.012745068" Sep 10 02:42:11.773895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:11.773856 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6k4pl"] Sep 10 02:42:11.774345 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:11.774070 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-6k4pl" podUID="eaa41136-e9ec-4902-964f-4703ae1b76e1" containerName="authorino" containerID="cri-o://78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021" gracePeriod=30 Sep 10 02:42:12.031457 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.031431 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6k4pl" Sep 10 02:42:12.122343 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.122309 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpgx6\" (UniqueName: \"kubernetes.io/projected/eaa41136-e9ec-4902-964f-4703ae1b76e1-kube-api-access-tpgx6\") pod \"eaa41136-e9ec-4902-964f-4703ae1b76e1\" (UID: \"eaa41136-e9ec-4902-964f-4703ae1b76e1\") " Sep 10 02:42:12.124365 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.124341 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa41136-e9ec-4902-964f-4703ae1b76e1-kube-api-access-tpgx6" (OuterVolumeSpecName: "kube-api-access-tpgx6") pod "eaa41136-e9ec-4902-964f-4703ae1b76e1" (UID: "eaa41136-e9ec-4902-964f-4703ae1b76e1"). InnerVolumeSpecName "kube-api-access-tpgx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 02:42:12.222964 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.222922 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpgx6\" (UniqueName: \"kubernetes.io/projected/eaa41136-e9ec-4902-964f-4703ae1b76e1-kube-api-access-tpgx6\") on node \"ip-10-0-143-228.ec2.internal\" DevicePath \"\"" Sep 10 02:42:12.732469 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.732424 2570 generic.go:356] "Generic (PLEG): container finished" podID="eaa41136-e9ec-4902-964f-4703ae1b76e1" containerID="78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021" exitCode=0 Sep 10 02:42:12.732672 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.732527 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6k4pl" Sep 10 02:42:12.732672 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.732528 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6k4pl" event={"ID":"eaa41136-e9ec-4902-964f-4703ae1b76e1","Type":"ContainerDied","Data":"78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021"} Sep 10 02:42:12.732672 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.732578 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6k4pl" event={"ID":"eaa41136-e9ec-4902-964f-4703ae1b76e1","Type":"ContainerDied","Data":"1c10f06f90c312b5ea0604d01917e62a3bfcc057d067efea6675832ad7e2e086"} Sep 10 02:42:12.732672 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.732601 2570 scope.go:117] "RemoveContainer" containerID="78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021" Sep 10 02:42:12.741396 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.741374 2570 scope.go:117] "RemoveContainer" containerID="78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021" Sep 10 02:42:12.741674 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:42:12.741653 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021\": container with ID starting with 78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021 not found: ID does not exist" containerID="78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021" Sep 10 02:42:12.741742 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.741687 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021"} err="failed to get container status \"78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021\": rpc error: code = NotFound desc = could not find container \"78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021\": container with ID starting with 78150879d4865bb7f059a7333bd5937c9f96b750bd39d0442a2650a360607021 not found: ID does not exist" Sep 10 02:42:12.754917 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.754893 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6k4pl"] Sep 10 02:42:12.758839 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:12.758798 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6k4pl"] Sep 10 02:42:13.227698 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:13.227665 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaa41136-e9ec-4902-964f-4703ae1b76e1" path="/var/lib/kubelet/pods/eaa41136-e9ec-4902-964f-4703ae1b76e1/volumes" Sep 10 02:42:17.383204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:17.383163 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:17.531899 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:17.531859 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:17.604847 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:17.604797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:17.795252 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:17.795217 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:18.539958 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:18.539910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:22.414671 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:22.414626 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:22.492152 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:22.492118 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:34.154268 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:34.154231 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:53.812783 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:53.812741 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:53.961573 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:53.961541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:55.204983 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.204939 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:55.305940 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.305904 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:55.350727 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.350692 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv"] Sep 10 02:42:55.351146 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.351128 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa41136-e9ec-4902-964f-4703ae1b76e1" containerName="authorino" Sep 10 02:42:55.351231 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.351148 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa41136-e9ec-4902-964f-4703ae1b76e1" containerName="authorino" Sep 10 02:42:55.351286 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.351275 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa41136-e9ec-4902-964f-4703ae1b76e1" containerName="authorino" Sep 10 02:42:55.357733 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.357711 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.360412 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.360382 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 02:42:55.360671 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.360652 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 02:42:55.360744 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.360669 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rt2j2\"" Sep 10 02:42:55.366365 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.366339 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv"] Sep 10 02:42:55.384521 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.384471 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-maas-api-tls\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.384701 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.384585 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb448749-9dcd-43f4-a75a-2130ced4e93c-openshift-service-ca\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.384701 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.384634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbwzg\" (UniqueName: \"kubernetes.io/projected/eb448749-9dcd-43f4-a75a-2130ced4e93c-kube-api-access-bbwzg\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.384701 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.384669 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-metrics-certs\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.485190 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.485106 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb448749-9dcd-43f4-a75a-2130ced4e93c-openshift-service-ca\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.485190 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.485160 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbwzg\" (UniqueName: \"kubernetes.io/projected/eb448749-9dcd-43f4-a75a-2130ced4e93c-kube-api-access-bbwzg\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.485417 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.485193 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-metrics-certs\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.485417 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.485231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-maas-api-tls\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.485417 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:42:55.485348 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 10 02:42:55.485555 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:42:55.485431 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-metrics-certs podName:eb448749-9dcd-43f4-a75a-2130ced4e93c nodeName:}" failed. No retries permitted until 2026-09-10 02:42:55.985408385 +0000 UTC m=+585.251504759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-metrics-certs") pod "maas-api-7d8f8fcfc6-p5ngv" (UID: "eb448749-9dcd-43f4-a75a-2130ced4e93c") : secret "maas-api-metrics-tls" not found Sep 10 02:42:55.485555 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:42:55.485355 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Sep 10 02:42:55.485668 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:42:55.485573 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-maas-api-tls podName:eb448749-9dcd-43f4-a75a-2130ced4e93c nodeName:}" failed. No retries permitted until 2026-09-10 02:42:55.985562 +0000 UTC m=+585.251658375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-maas-api-tls") pod "maas-api-7d8f8fcfc6-p5ngv" (UID: "eb448749-9dcd-43f4-a75a-2130ced4e93c") : secret "maas-api-serving-cert" not found Sep 10 02:42:55.485794 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.485776 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb448749-9dcd-43f4-a75a-2130ced4e93c-openshift-service-ca\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.495324 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.495295 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbwzg\" (UniqueName: \"kubernetes.io/projected/eb448749-9dcd-43f4-a75a-2130ced4e93c-kube-api-access-bbwzg\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.600411 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.600355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:55.703009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.702965 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:55.800793 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.800700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:55.872782 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.872743 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:42:55.878307 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.878275 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-99bf5"] Sep 10 02:42:55.881898 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.881878 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:55.885872 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.885841 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 02:42:55.885872 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.885820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l9t6z\"" Sep 10 02:42:55.898057 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.898026 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-99bf5"] Sep 10 02:42:55.990704 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.990663 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07416bc-6df0-4ddb-acb3-d64eec3d40a9-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-99bf5\" (UID: \"d07416bc-6df0-4ddb-acb3-d64eec3d40a9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:55.990928 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.990735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-metrics-certs\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.990928 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.990765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-maas-api-tls\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.990928 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.990784 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwmc\" (UniqueName: \"kubernetes.io/projected/d07416bc-6df0-4ddb-acb3-d64eec3d40a9-kube-api-access-6bwmc\") pod \"payload-processing-5b54b7fbb7-99bf5\" (UID: \"d07416bc-6df0-4ddb-acb3-d64eec3d40a9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:55.993398 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.993367 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-maas-api-tls\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:55.993840 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:55.993801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb448749-9dcd-43f4-a75a-2130ced4e93c-metrics-certs\") pod \"maas-api-7d8f8fcfc6-p5ngv\" (UID: \"eb448749-9dcd-43f4-a75a-2130ced4e93c\") " pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:56.091764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.091669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07416bc-6df0-4ddb-acb3-d64eec3d40a9-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-99bf5\" (UID: \"d07416bc-6df0-4ddb-acb3-d64eec3d40a9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:56.091764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.091738 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwmc\" (UniqueName: \"kubernetes.io/projected/d07416bc-6df0-4ddb-acb3-d64eec3d40a9-kube-api-access-6bwmc\") pod \"payload-processing-5b54b7fbb7-99bf5\" (UID: \"d07416bc-6df0-4ddb-acb3-d64eec3d40a9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:56.092594 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.092563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d07416bc-6df0-4ddb-acb3-d64eec3d40a9-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-99bf5\" (UID: \"d07416bc-6df0-4ddb-acb3-d64eec3d40a9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:56.101100 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.101075 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwmc\" (UniqueName: \"kubernetes.io/projected/d07416bc-6df0-4ddb-acb3-d64eec3d40a9-kube-api-access-6bwmc\") pod \"payload-processing-5b54b7fbb7-99bf5\" (UID: \"d07416bc-6df0-4ddb-acb3-d64eec3d40a9\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:56.193476 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.193434 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:42:56.270550 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.270500 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:42:56.474410 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.474386 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-99bf5"] Sep 10 02:42:56.476286 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:42:56.476264 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07416bc_6df0_4ddb_acb3_d64eec3d40a9.slice/crio-03adb69ec0712f1788a9493e4826527a5251d4a1378829788cd0be1cadd7719a WatchSource:0}: Error finding container 03adb69ec0712f1788a9493e4826527a5251d4a1378829788cd0be1cadd7719a: Status 404 returned error can't find the container with id 03adb69ec0712f1788a9493e4826527a5251d4a1378829788cd0be1cadd7719a Sep 10 02:42:56.544928 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.544903 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv"] Sep 10 02:42:56.546966 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:42:56.546940 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb448749_9dcd_43f4_a75a_2130ced4e93c.slice/crio-43d088e809c41687858df0a8a29266862a08a278819a801edbbcc4de7a0bd932 WatchSource:0}: Error finding container 43d088e809c41687858df0a8a29266862a08a278819a801edbbcc4de7a0bd932: Status 404 returned error can't find the container with id 43d088e809c41687858df0a8a29266862a08a278819a801edbbcc4de7a0bd932 Sep 10 02:42:56.946118 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.946080 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" event={"ID":"eb448749-9dcd-43f4-a75a-2130ced4e93c","Type":"ContainerStarted","Data":"43d088e809c41687858df0a8a29266862a08a278819a801edbbcc4de7a0bd932"} Sep 10 02:42:56.947296 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:42:56.947270 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" event={"ID":"d07416bc-6df0-4ddb-acb3-d64eec3d40a9","Type":"ContainerStarted","Data":"03adb69ec0712f1788a9493e4826527a5251d4a1378829788cd0be1cadd7719a"} Sep 10 02:43:02.000983 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:02.000937 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" event={"ID":"eb448749-9dcd-43f4-a75a-2130ced4e93c","Type":"ContainerStarted","Data":"00c16805fa092f63323c1229ae84ea415d5a3286e7a7026fdcf351cc1b5bce57"} Sep 10 02:43:02.001457 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:02.001007 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:43:02.002637 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:02.002615 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" event={"ID":"d07416bc-6df0-4ddb-acb3-d64eec3d40a9","Type":"ContainerStarted","Data":"070074b9fa81e6de8ff60ba7c026607064768ad2dc89dc05af993664e9a7cef1"} Sep 10 02:43:02.002724 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:02.002696 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:43:02.019373 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:02.019318 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" podStartSLOduration=2.436333106 podStartE2EDuration="7.019302975s" totalImagesPullingTime="4.582969869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:43:02.018340273 +0000 UTC m=+591.284436668" watchObservedRunningTime="2026-09-10 02:43:02.019302975 +0000 UTC m=+591.285399366" Sep 10 02:43:06.029798 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:06.029758 2570 generic.go:356] "Generic (PLEG): container finished" podID="77a55644-04be-4f11-b5a5-ff85419e3bfe" containerID="a49a25034bc46f93585d9cbb5a2a7603cc4721a01cce7c3426803c720db729f3" exitCode=0 Sep 10 02:43:06.030236 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:06.029841 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" event={"ID":"77a55644-04be-4f11-b5a5-ff85419e3bfe","Type":"ContainerDied","Data":"a49a25034bc46f93585d9cbb5a2a7603cc4721a01cce7c3426803c720db729f3"} Sep 10 02:43:06.030297 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:06.030236 2570 scope.go:117] "RemoveContainer" containerID="a49a25034bc46f93585d9cbb5a2a7603cc4721a01cce7c3426803c720db729f3" Sep 10 02:43:06.031174 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:06.031145 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:43:06.031236 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:06.031209 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:43:06.031276 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:06.031241 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 02:43:06.051149 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:06.051099 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" podStartSLOduration=6.329093701 podStartE2EDuration="11.051081487s" totalImagesPullingTime="4.721987786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 02:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:43:02.034986144 +0000 UTC m=+591.301082549" watchObservedRunningTime="2026-09-10 02:43:06.051081487 +0000 UTC m=+595.317177877" Sep 10 02:43:07.036754 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:07.036721 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" event={"ID":"77a55644-04be-4f11-b5a5-ff85419e3bfe","Type":"ContainerStarted","Data":"f7397ac903b3433b744a8d2401fdd67502b6e891d552284b3178a45b412f7458"} Sep 10 02:43:07.242259 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:07.242225 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:43:07.243409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:07.243382 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gfpvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 10 02:43:07.243520 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:07.243431 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" podUID="77a55644-04be-4f11-b5a5-ff85419e3bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 10 02:43:08.013482 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:08.013453 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d8f8fcfc6-p5ngv" Sep 10 02:43:08.242199 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:08.242159 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gfpvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 10 02:43:08.242574 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:08.242217 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" podUID="77a55644-04be-4f11-b5a5-ff85419e3bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 10 02:43:09.094353 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:09.094317 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:09.193225 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:09.193189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:09.242669 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:09.242637 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gfpvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 10 02:43:09.243047 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:09.242690 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" podUID="77a55644-04be-4f11-b5a5-ff85419e3bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 10 02:43:09.542683 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:09.542645 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:43:10.242472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:10.242435 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gfpvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 10 02:43:10.242658 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:10.242502 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" podUID="77a55644-04be-4f11-b5a5-ff85419e3bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 10 02:43:11.198647 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:11.198622 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:43:11.202965 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:11.200377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:43:11.242009 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:11.241982 2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-gfpvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Sep 10 02:43:11.242147 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:11.242031 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" podUID="77a55644-04be-4f11-b5a5-ff85419e3bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Sep 10 02:43:12.246541 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:12.246511 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:43:12.247463 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:12.247446 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-gfpvz" Sep 10 02:43:13.010221 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:13.010188 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-99bf5" Sep 10 02:43:14.449160 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:14.449129 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:17.410080 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.410043 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk"] Sep 10 02:43:17.417894 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.417861 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.421854 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.421615 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g7ctp\"" Sep 10 02:43:17.421854 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.421658 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 02:43:17.422058 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.421927 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 02:43:17.422058 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.421961 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 02:43:17.423784 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.423760 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk"] Sep 10 02:43:17.436316 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.436286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:17.483460 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.483423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.483630 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.483471 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.483630 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.483492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b89a6d0-864b-4a27-8fa6-cce036e4425d-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.483630 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.483509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkf28\" (UniqueName: \"kubernetes.io/projected/3b89a6d0-864b-4a27-8fa6-cce036e4425d-kube-api-access-tkf28\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.483630 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.483533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.483630 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.483557 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584139 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b89a6d0-864b-4a27-8fa6-cce036e4425d-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkf28\" (UniqueName: \"kubernetes.io/projected/3b89a6d0-864b-4a27-8fa6-cce036e4425d-kube-api-access-tkf28\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584380 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584608 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584739 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584708 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.584739 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.584721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.586677 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.586654 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b89a6d0-864b-4a27-8fa6-cce036e4425d-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.586882 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.586858 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b89a6d0-864b-4a27-8fa6-cce036e4425d-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.591726 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.591703 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkf28\" (UniqueName: \"kubernetes.io/projected/3b89a6d0-864b-4a27-8fa6-cce036e4425d-kube-api-access-tkf28\") pod \"e2e-distinct-simulated-kserve-6685d86975-d7mzk\" (UID: \"3b89a6d0-864b-4a27-8fa6-cce036e4425d\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:17.729817 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:17.729777 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:18.022244 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:18.022219 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk"] Sep 10 02:43:18.024437 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:43:18.024411 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b89a6d0_864b_4a27_8fa6_cce036e4425d.slice/crio-ab8d69fb48b383ed1cc989cb5d460e0a5f29c42f0a45dd3063b6ba2ef438d876 WatchSource:0}: Error finding container ab8d69fb48b383ed1cc989cb5d460e0a5f29c42f0a45dd3063b6ba2ef438d876: Status 404 returned error can't find the container with id ab8d69fb48b383ed1cc989cb5d460e0a5f29c42f0a45dd3063b6ba2ef438d876 Sep 10 02:43:18.093889 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:18.093848 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" event={"ID":"3b89a6d0-864b-4a27-8fa6-cce036e4425d","Type":"ContainerStarted","Data":"ab8d69fb48b383ed1cc989cb5d460e0a5f29c42f0a45dd3063b6ba2ef438d876"} Sep 10 02:43:20.105953 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:20.105919 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" containerID="6f99cef7f1b3d3b858f530fa603af747b54ede70526a3e001ee4d669f2b14426" exitCode=0 Sep 10 02:43:20.106327 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:20.105999 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" event={"ID":"3b89a6d0-864b-4a27-8fa6-cce036e4425d","Type":"ContainerDied","Data":"6f99cef7f1b3d3b858f530fa603af747b54ede70526a3e001ee4d669f2b14426"} Sep 10 02:43:20.106327 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:20.106187 2570 scope.go:117] "RemoveContainer" containerID="6f99cef7f1b3d3b858f530fa603af747b54ede70526a3e001ee4d669f2b14426" Sep 10 02:43:21.112596 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:21.112562 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" containerID="40f92ba094746c18cafbd284f8f8a7384ea1b8326f882d55389c02ee23047931" exitCode=0 Sep 10 02:43:21.113054 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:21.112648 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" event={"ID":"3b89a6d0-864b-4a27-8fa6-cce036e4425d","Type":"ContainerDied","Data":"40f92ba094746c18cafbd284f8f8a7384ea1b8326f882d55389c02ee23047931"} Sep 10 02:43:21.113054 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:21.112705 2570 scope.go:117] "RemoveContainer" containerID="6f99cef7f1b3d3b858f530fa603af747b54ede70526a3e001ee4d669f2b14426" Sep 10 02:43:21.113054 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:21.113018 2570 scope.go:117] "RemoveContainer" containerID="40f92ba094746c18cafbd284f8f8a7384ea1b8326f882d55389c02ee23047931" Sep 10 02:43:21.113212 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:21.113193 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:43:21.938947 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:21.938908 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:22.119442 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:22.119406 2570 scope.go:117] "RemoveContainer" containerID="40f92ba094746c18cafbd284f8f8a7384ea1b8326f882d55389c02ee23047931" Sep 10 02:43:22.119847 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:22.119628 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:43:27.730912 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:27.730867 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:27.731304 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:27.730934 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:27.731304 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:27.730950 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:27.731304 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:27.730962 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:27.731409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:27.731331 2570 scope.go:117] "RemoveContainer" containerID="40f92ba094746c18cafbd284f8f8a7384ea1b8326f882d55389c02ee23047931" Sep 10 02:43:27.731523 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:27.731503 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:43:34.992280 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:34.992187 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:35.338759 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:35.338669 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:43.222586 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:43.222553 2570 scope.go:117] "RemoveContainer" containerID="40f92ba094746c18cafbd284f8f8a7384ea1b8326f882d55389c02ee23047931" Sep 10 02:43:43.548013 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:43.547974 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:44.231803 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:44.231771 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" containerID="23eed7ba052137ddc858188cb887eb9f29ffae1c20a6ea7884583d1d89266181" exitCode=0 Sep 10 02:43:44.232194 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:44.231854 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" event={"ID":"3b89a6d0-864b-4a27-8fa6-cce036e4425d","Type":"ContainerDied","Data":"23eed7ba052137ddc858188cb887eb9f29ffae1c20a6ea7884583d1d89266181"} Sep 10 02:43:44.232194 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:44.231902 2570 scope.go:117] "RemoveContainer" containerID="40f92ba094746c18cafbd284f8f8a7384ea1b8326f882d55389c02ee23047931" Sep 10 02:43:44.232194 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:44.232137 2570 scope.go:117] "RemoveContainer" containerID="23eed7ba052137ddc858188cb887eb9f29ffae1c20a6ea7884583d1d89266181" Sep 10 02:43:44.232339 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:44.232297 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:43:47.730638 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:47.730594 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:47.731072 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:47.730664 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:47.731072 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:47.730682 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:47.731072 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:47.730694 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:43:47.731179 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:47.731076 2570 scope.go:117] "RemoveContainer" containerID="23eed7ba052137ddc858188cb887eb9f29ffae1c20a6ea7884583d1d89266181" Sep 10 02:43:47.731257 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:47.731238 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:43:50.008344 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.008310 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47"] Sep 10 02:43:50.014333 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.014312 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.016791 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.016753 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 02:43:50.020763 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.020735 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47"] Sep 10 02:43:50.041764 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.041730 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:50.075156 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.075119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e0a214-7b96-4a60-97c1-9e3619b9c00a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.075156 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.075155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nkx\" (UniqueName: \"kubernetes.io/projected/78e0a214-7b96-4a60-97c1-9e3619b9c00a-kube-api-access-59nkx\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.075359 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.075220 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.075359 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.075273 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.075359 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.075294 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.075359 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.075331 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.176844 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.176786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177029 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.176850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177029 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.176891 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177029 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.176985 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e0a214-7b96-4a60-97c1-9e3619b9c00a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177029 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.177015 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nkx\" (UniqueName: \"kubernetes.io/projected/78e0a214-7b96-4a60-97c1-9e3619b9c00a-kube-api-access-59nkx\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177258 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.177067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177258 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.177236 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177366 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.177304 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.177457 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.177420 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.179409 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.179381 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e0a214-7b96-4a60-97c1-9e3619b9c00a-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.179959 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.179939 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e0a214-7b96-4a60-97c1-9e3619b9c00a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.184871 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.184849 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nkx\" (UniqueName: \"kubernetes.io/projected/78e0a214-7b96-4a60-97c1-9e3619b9c00a-kube-api-access-59nkx\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47\" (UID: \"78e0a214-7b96-4a60-97c1-9e3619b9c00a\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.327188 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.327099 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:43:50.602025 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.601999 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47"] Sep 10 02:43:50.603774 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:43:50.603747 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e0a214_7b96_4a60_97c1_9e3619b9c00a.slice/crio-491fe5bee7b3c4b33d3947d92debbe197a92ce9de8af2bc5dd26213e2c616510 WatchSource:0}: Error finding container 491fe5bee7b3c4b33d3947d92debbe197a92ce9de8af2bc5dd26213e2c616510: Status 404 returned error can't find the container with id 491fe5bee7b3c4b33d3947d92debbe197a92ce9de8af2bc5dd26213e2c616510 Sep 10 02:43:50.605123 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:50.605104 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 02:43:51.272152 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:51.272116 2570 generic.go:356] "Generic (PLEG): container finished" podID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" containerID="5f5d3226ecb656de54b2dc7b04b06b31a9ada9cac1eb536b91bda390514df3c7" exitCode=0 Sep 10 02:43:51.272617 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:51.272199 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" event={"ID":"78e0a214-7b96-4a60-97c1-9e3619b9c00a","Type":"ContainerDied","Data":"5f5d3226ecb656de54b2dc7b04b06b31a9ada9cac1eb536b91bda390514df3c7"} Sep 10 02:43:51.272617 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:51.272252 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" event={"ID":"78e0a214-7b96-4a60-97c1-9e3619b9c00a","Type":"ContainerStarted","Data":"491fe5bee7b3c4b33d3947d92debbe197a92ce9de8af2bc5dd26213e2c616510"} Sep 10 02:43:51.272617 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:51.272410 2570 scope.go:117] "RemoveContainer" containerID="5f5d3226ecb656de54b2dc7b04b06b31a9ada9cac1eb536b91bda390514df3c7" Sep 10 02:43:52.280750 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:52.280707 2570 generic.go:356] "Generic (PLEG): container finished" podID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" containerID="b664b0f63a7261191e0c175ee3502cefab0aa6d78e2c797240603dbb1595c14a" exitCode=0 Sep 10 02:43:52.281224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:52.280778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" event={"ID":"78e0a214-7b96-4a60-97c1-9e3619b9c00a","Type":"ContainerDied","Data":"b664b0f63a7261191e0c175ee3502cefab0aa6d78e2c797240603dbb1595c14a"} Sep 10 02:43:52.281224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:52.280865 2570 scope.go:117] "RemoveContainer" containerID="5f5d3226ecb656de54b2dc7b04b06b31a9ada9cac1eb536b91bda390514df3c7" Sep 10 02:43:52.281224 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:52.281153 2570 scope.go:117] "RemoveContainer" containerID="b664b0f63a7261191e0c175ee3502cefab0aa6d78e2c797240603dbb1595c14a" Sep 10 02:43:52.281439 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:52.281412 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:43:53.287377 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:53.287348 2570 scope.go:117] "RemoveContainer" containerID="b664b0f63a7261191e0c175ee3502cefab0aa6d78e2c797240603dbb1595c14a" Sep 10 02:43:53.287744 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:53.287520 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:43:57.037138 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:57.037106 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:43:58.222574 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:43:58.222543 2570 scope.go:117] "RemoveContainer" containerID="23eed7ba052137ddc858188cb887eb9f29ffae1c20a6ea7884583d1d89266181" Sep 10 02:43:58.223006 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:43:58.222726 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:44:00.327887 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:00.327854 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:00.328278 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:00.327911 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:00.328278 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:00.327926 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:00.328278 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:00.327939 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:00.328384 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:00.328332 2570 scope.go:117] "RemoveContainer" containerID="b664b0f63a7261191e0c175ee3502cefab0aa6d78e2c797240603dbb1595c14a" Sep 10 02:44:00.328513 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:00.328493 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:44:09.222503 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:09.222465 2570 scope.go:117] "RemoveContainer" containerID="23eed7ba052137ddc858188cb887eb9f29ffae1c20a6ea7884583d1d89266181" Sep 10 02:44:10.372908 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:10.372872 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" exitCode=0 Sep 10 02:44:10.373288 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:10.372938 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" event={"ID":"3b89a6d0-864b-4a27-8fa6-cce036e4425d","Type":"ContainerDied","Data":"e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91"} Sep 10 02:44:10.373288 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:10.373021 2570 scope.go:117] "RemoveContainer" containerID="23eed7ba052137ddc858188cb887eb9f29ffae1c20a6ea7884583d1d89266181" Sep 10 02:44:10.373388 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:10.373343 2570 scope.go:117] "RemoveContainer" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" Sep 10 02:44:10.373587 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:10.373569 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:44:15.221959 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:15.221918 2570 scope.go:117] "RemoveContainer" containerID="b664b0f63a7261191e0c175ee3502cefab0aa6d78e2c797240603dbb1595c14a" Sep 10 02:44:16.409642 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:16.409610 2570 generic.go:356] "Generic (PLEG): container finished" podID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" containerID="991dc36a31f38954cac9a871ce823681de7ccc3cf2c3426fa0b15b97fb828081" exitCode=0 Sep 10 02:44:16.410052 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:16.409661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" event={"ID":"78e0a214-7b96-4a60-97c1-9e3619b9c00a","Type":"ContainerDied","Data":"991dc36a31f38954cac9a871ce823681de7ccc3cf2c3426fa0b15b97fb828081"} Sep 10 02:44:16.410052 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:16.409714 2570 scope.go:117] "RemoveContainer" containerID="b664b0f63a7261191e0c175ee3502cefab0aa6d78e2c797240603dbb1595c14a" Sep 10 02:44:16.410148 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:16.410081 2570 scope.go:117] "RemoveContainer" containerID="991dc36a31f38954cac9a871ce823681de7ccc3cf2c3426fa0b15b97fb828081" Sep 10 02:44:16.410289 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:16.410269 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:44:17.730937 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:17.730905 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:17.731336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:17.730967 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:17.731336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:17.730983 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:17.731336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:17.730999 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:17.731457 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:17.731339 2570 scope.go:117] "RemoveContainer" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" Sep 10 02:44:17.731522 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:17.731503 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:44:18.421493 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:18.421464 2570 scope.go:117] "RemoveContainer" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" Sep 10 02:44:18.421677 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:18.421658 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:44:20.328010 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:20.327972 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:20.328393 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:20.328060 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:20.328393 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:20.328077 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:20.328393 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:20.328090 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:44:20.328501 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:20.328419 2570 scope.go:117] "RemoveContainer" containerID="991dc36a31f38954cac9a871ce823681de7ccc3cf2c3426fa0b15b97fb828081" Sep 10 02:44:20.328609 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:20.328588 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:44:31.225323 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:31.225292 2570 scope.go:117] "RemoveContainer" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" Sep 10 02:44:31.225790 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:31.225484 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:44:35.222191 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:35.222156 2570 scope.go:117] "RemoveContainer" containerID="991dc36a31f38954cac9a871ce823681de7ccc3cf2c3426fa0b15b97fb828081" Sep 10 02:44:35.222587 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:35.222334 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:44:42.222179 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:42.222147 2570 scope.go:117] "RemoveContainer" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" Sep 10 02:44:42.222630 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:42.222332 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:44:50.222430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:50.222396 2570 scope.go:117] "RemoveContainer" containerID="991dc36a31f38954cac9a871ce823681de7ccc3cf2c3426fa0b15b97fb828081" Sep 10 02:44:50.584194 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:50.584167 2570 generic.go:356] "Generic (PLEG): container finished" podID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" exitCode=0 Sep 10 02:44:50.584321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:50.584244 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" event={"ID":"78e0a214-7b96-4a60-97c1-9e3619b9c00a","Type":"ContainerDied","Data":"88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937"} Sep 10 02:44:50.584321 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:50.584301 2570 scope.go:117] "RemoveContainer" containerID="991dc36a31f38954cac9a871ce823681de7ccc3cf2c3426fa0b15b97fb828081" Sep 10 02:44:50.584663 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:50.584647 2570 scope.go:117] "RemoveContainer" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" Sep 10 02:44:50.584888 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:50.584865 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:44:53.222312 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:53.222281 2570 scope.go:117] "RemoveContainer" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" Sep 10 02:44:53.601415 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:53.601388 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" exitCode=0 Sep 10 02:44:53.601564 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:53.601444 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" event={"ID":"3b89a6d0-864b-4a27-8fa6-cce036e4425d","Type":"ContainerDied","Data":"a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b"} Sep 10 02:44:53.601564 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:53.601524 2570 scope.go:117] "RemoveContainer" containerID="e3f47f18c77766fd43842cdac409ac232c9e2e8e1f824c75edbbe25c8b281b91" Sep 10 02:44:53.601913 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:53.601894 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:44:53.602126 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:53.602105 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:44:57.730905 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:57.730868 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:57.731305 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:57.730935 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:57.731305 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:57.730955 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:57.731305 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:57.730971 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:44:57.731414 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:44:57.731336 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:44:57.731518 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:44:57.731501 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:45:00.327642 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:00.327596 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:00.328102 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:00.327668 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:00.328102 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:00.327687 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:00.328102 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:00.327701 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:00.328102 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:00.328050 2570 scope.go:117] "RemoveContainer" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" Sep 10 02:45:00.328265 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:00.328245 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:45:00.637430 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:00.637354 2570 scope.go:117] "RemoveContainer" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" Sep 10 02:45:00.637570 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:00.637535 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:45:11.224191 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:11.224110 2570 scope.go:117] "RemoveContainer" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" Sep 10 02:45:11.224553 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:11.224389 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:45:12.221989 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:12.221959 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:45:12.222177 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:12.222156 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:45:24.014377 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:24.014332 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:45:26.222153 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:26.222122 2570 scope.go:117] "RemoveContainer" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" Sep 10 02:45:26.222512 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:26.222295 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:45:27.222331 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:27.222296 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:45:27.222703 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:27.222512 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:45:39.222180 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:39.222139 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:45:39.222665 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:39.222352 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:45:41.224265 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:41.224237 2570 scope.go:117] "RemoveContainer" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" Sep 10 02:45:41.834271 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:41.834174 2570 generic.go:356] "Generic (PLEG): container finished" podID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" exitCode=0 Sep 10 02:45:41.834271 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:41.834238 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" event={"ID":"78e0a214-7b96-4a60-97c1-9e3619b9c00a","Type":"ContainerDied","Data":"dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b"} Sep 10 02:45:41.834511 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:41.834296 2570 scope.go:117] "RemoveContainer" containerID="88e93e42399f86945eaa37b7ad7ac809d9491b44b5d48b95b133e986e082d937" Sep 10 02:45:41.834627 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:41.834611 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:45:41.834812 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:41.834794 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:45:50.327994 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:50.327959 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:50.328385 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:50.328025 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:50.328385 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:50.328041 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:50.328385 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:50.328053 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:45:50.328496 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:50.328408 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:45:50.328603 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:50.328583 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:45:54.222749 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:54.222711 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:45:54.223240 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:45:54.222972 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:45:54.408715 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:54.408674 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:45:54.461424 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:45:54.461390 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:46:04.222298 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:04.222264 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:46:04.222762 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:04.222429 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:46:07.222186 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:07.222157 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:46:07.222568 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:07.222299 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:46:15.222565 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:15.222529 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:46:15.222985 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:15.222721 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:46:21.225049 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:21.225021 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:46:22.038232 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:22.038197 2570 generic.go:356] "Generic (PLEG): container finished" podID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" exitCode=0 Sep 10 02:46:22.038432 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:22.038244 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" event={"ID":"3b89a6d0-864b-4a27-8fa6-cce036e4425d","Type":"ContainerDied","Data":"fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed"} Sep 10 02:46:22.038432 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:22.038334 2570 scope.go:117] "RemoveContainer" containerID="a07a676ffd78ec6ebd4922e19091ea2f1e7ad6bd4ed6c50897a58d82d4f13b6b" Sep 10 02:46:22.038741 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:22.038719 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:46:22.038945 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:22.038923 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:46:27.730204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:27.730166 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:46:27.730618 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:27.730228 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:46:27.730618 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:27.730245 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:46:27.730618 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:27.730257 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" Sep 10 02:46:27.730618 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:27.730596 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:46:27.730778 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:27.730759 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:46:30.222131 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:30.222085 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:46:30.222599 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:30.222343 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:46:42.222556 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:42.222480 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:46:42.223021 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:42.222661 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:46:44.221949 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:44.221916 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:46:44.222333 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:44.222121 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:46:55.221906 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:55.221876 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:46:55.222354 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:46:55.221961 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:46:55.222354 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:55.222062 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:46:55.222354 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:46:55.222167 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:47:07.222052 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:07.222016 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:47:08.275187 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:08.275149 2570 generic.go:356] "Generic (PLEG): container finished" podID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" exitCode=0 Sep 10 02:47:08.275592 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:08.275219 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" event={"ID":"78e0a214-7b96-4a60-97c1-9e3619b9c00a","Type":"ContainerDied","Data":"30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b"} Sep 10 02:47:08.275592 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:08.275272 2570 scope.go:117] "RemoveContainer" containerID="dc34c1b79ac1c17141b1bccbc56a726f047cebdd54fc3a1faa1dd2df922b594b" Sep 10 02:47:08.275685 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:08.275639 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:47:08.275858 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:08.275837 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:47:09.222507 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:09.222478 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:47:09.222709 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:09.222667 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:47:10.327336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:10.327298 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:47:10.327717 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:10.327364 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:47:10.327717 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:10.327533 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:47:10.327717 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:10.327652 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" Sep 10 02:47:10.327868 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:10.327755 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:47:10.327986 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:10.327967 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:47:11.292204 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:11.292174 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:47:11.292373 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:11.292357 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:47:22.222513 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:22.222475 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:47:22.223050 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:22.222720 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:47:26.222460 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:26.222429 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:47:26.222856 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:26.222585 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:47:37.221991 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:37.221959 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:47:37.222353 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:37.222148 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:47:39.222041 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:39.222006 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:47:39.222438 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:39.222212 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:47:51.224027 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:51.223994 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:47:51.224498 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:51.224219 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:47:51.224498 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:47:51.224350 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:47:51.224629 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:47:51.224530 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:48:03.222256 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:03.222216 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:48:03.222798 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:03.222457 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:48:04.222228 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:04.222199 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:48:04.222401 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:04.222380 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:48:11.240646 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:11.240620 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:48:11.243149 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:11.242481 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:48:12.717656 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:12.717620 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_78e0a214-7b96-4a60-97c1-9e3619b9c00a/main/5.log" Sep 10 02:48:12.839274 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:12.839249 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_78e0a214-7b96-4a60-97c1-9e3619b9c00a/main/5.log" Sep 10 02:48:13.990347 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:13.990316 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c574fbcf-b97jg_ddc0c832-e730-4741-9b9d-2d0b3e182a95/authorino/0.log" Sep 10 02:48:18.222620 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:18.222590 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:48:18.223080 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:18.222676 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:48:18.223080 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:18.222822 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:48:18.223080 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:18.222860 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:48:19.549785 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:19.549754 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pztgq_a94084da-440c-4184-9d87-6447ff838e10/manager/0.log" Sep 10 02:48:19.675982 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:19.675948 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gc4th_49019557-6a1c-4039-ac2f-78844c56798d/manager/0.log" Sep 10 02:48:20.469106 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:20.469078 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647784d6dd-s9mvw_e73accdd-c852-4fd5-b4b1-747844cd7335/copy-manifests/0.log" Sep 10 02:48:20.476278 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:20.476253 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647784d6dd-s9mvw_e73accdd-c852-4fd5-b4b1-747844cd7335/manager/0.log" Sep 10 02:48:20.820519 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:20.820433 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7d8f8fcfc6-p5ngv_eb448749-9dcd-43f4-a75a-2130ced4e93c/maas-api/0.log" Sep 10 02:48:21.045651 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:21.045618 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gzhrc_ed3a2902-4b84-47c2-b44c-8c17d0e86d24/postgres/0.log" Sep 10 02:48:21.857220 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:21.857184 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c574fbcf-b97jg_ddc0c832-e730-4741-9b9d-2d0b3e182a95/authorino/0.log" Sep 10 02:48:21.973358 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:21.973325 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-v69vk_05b528f6-094d-4a0f-a11f-7c15cb8c0a58/manager/0.log" Sep 10 02:48:22.091677 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:22.091642 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-77t4g_4897925a-0ae0-4a75-9c4e-34857cbb1e99/manager/0.log" Sep 10 02:48:22.582292 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:22.582259 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-q2pp2_6c970b0b-1057-46ca-9c02-7819728eb71e/manager/0.log" Sep 10 02:48:23.323055 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:23.323007 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d6b6cf86-z6zzd_97bd02c0-6aeb-49c0-ba0c-3f4521345b6e/kube-auth-proxy/0.log" Sep 10 02:48:23.449659 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:23.449612 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gfpvz_77a55644-04be-4f11-b5a5-ff85419e3bfe/istio-proxy/1.log" Sep 10 02:48:23.681524 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:23.681499 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-99bf5_d07416bc-6df0-4ddb-acb3-d64eec3d40a9/payload-processing/0.log" Sep 10 02:48:23.802801 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:23.802777 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fb5dd5bbd-khhbd_d22aa7ab-c491-460d-8415-d452a6a64976/router/0.log" Sep 10 02:48:24.301712 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:24.301682 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-6685d86975-d7mzk_3b89a6d0-864b-4a27-8fa6-cce036e4425d/main/5.log" Sep 10 02:48:24.775735 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:24.775708 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_78e0a214-7b96-4a60-97c1-9e3619b9c00a/main/5.log" Sep 10 02:48:25.046549 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:25.046425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:48:25.099643 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:25.099610 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:48:30.417770 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:30.417731 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:48:30.519389 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:30.519355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:48:32.222069 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:32.222034 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:48:32.222460 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:32.222279 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:48:33.222296 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:33.222260 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:48:33.222691 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:33.222449 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:48:33.573517 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:33.573426 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sg8dz_3902b22c-0dda-4604-a7d8-439602edd83e/global-pull-secret-syncer/0.log" Sep 10 02:48:33.656237 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:33.656204 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" Sep 10 02:48:33.771438 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:33.771394 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-228.ec2.internal_728b04a33548859b274bbec2cc5b3fee/haproxy/0.log" Sep 10 02:48:38.053613 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:38.053583 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c574fbcf-b97jg_ddc0c832-e730-4741-9b9d-2d0b3e182a95/authorino/0.log" Sep 10 02:48:38.082839 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:38.082804 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-v69vk_05b528f6-094d-4a0f-a11f-7c15cb8c0a58/manager/0.log" Sep 10 02:48:38.114651 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:38.114625 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-77t4g_4897925a-0ae0-4a75-9c4e-34857cbb1e99/manager/0.log" Sep 10 02:48:38.250603 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:38.250569 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-q2pp2_6c970b0b-1057-46ca-9c02-7819728eb71e/manager/0.log" Sep 10 02:48:39.830850 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:39.830803 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-frrmp_ffbf8b54-3714-486c-bc13-4eeb9cd28d26/cluster-monitoring-operator/0.log" Sep 10 02:48:40.057142 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:40.057109 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgrfb_b18c3d5a-7b08-4dc2-88a1-a3e576394832/node-exporter/0.log" Sep 10 02:48:40.080514 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:40.080491 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgrfb_b18c3d5a-7b08-4dc2-88a1-a3e576394832/kube-rbac-proxy/0.log" Sep 10 02:48:40.104865 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:40.104812 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dgrfb_b18c3d5a-7b08-4dc2-88a1-a3e576394832/init-textfile/0.log" Sep 10 02:48:42.435140 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:42.435112 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-99qz4_a9f5105d-d773-45f9-afc2-69a1f0fbc218/networking-console-plugin/0.log" Sep 10 02:48:43.118174 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.118126 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/2.log" Sep 10 02:48:43.121895 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.121875 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-xr5ft_a2400c62-09d7-4e9a-9640-85ce68a09fb4/console-operator/3.log" Sep 10 02:48:43.970394 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.970356 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb"] Sep 10 02:48:43.973837 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.973808 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:43.977572 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.977548 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4dzq\"/\"openshift-service-ca.crt\"" Sep 10 02:48:43.977681 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.977574 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4dzq\"/\"kube-root-ca.crt\"" Sep 10 02:48:43.979217 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.979196 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m4dzq\"/\"default-dockercfg-9mkwm\"" Sep 10 02:48:43.986045 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:43.986024 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb"] Sep 10 02:48:44.048376 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.048339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4sxf\" (UniqueName: \"kubernetes.io/projected/b99ed8a0-751b-4e2e-a25d-7387de92aa04-kube-api-access-f4sxf\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.048546 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.048378 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-proc\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.048546 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.048458 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-podres\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.048546 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.048481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-sys\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.048546 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.048531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-lib-modules\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.149950 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.149915 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-lib-modules\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150119 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.149994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4sxf\" (UniqueName: \"kubernetes.io/projected/b99ed8a0-751b-4e2e-a25d-7387de92aa04-kube-api-access-f4sxf\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150119 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.150019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-proc\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150119 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.150057 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-podres\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150119 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.150086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-sys\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150265 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.150119 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-lib-modules\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150265 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.150133 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-proc\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150265 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.150183 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-sys\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.150265 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.150195 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b99ed8a0-751b-4e2e-a25d-7387de92aa04-podres\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.157462 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.157434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4sxf\" (UniqueName: \"kubernetes.io/projected/b99ed8a0-751b-4e2e-a25d-7387de92aa04-kube-api-access-f4sxf\") pod \"perf-node-gather-daemonset-l9shb\" (UID: \"b99ed8a0-751b-4e2e-a25d-7387de92aa04\") " pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.222253 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.222194 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:48:44.222363 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:44.222348 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:48:44.284273 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.284247 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.473775 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.473689 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-4c45p_41df8304-9f36-45be-a5c7-696eb836d8e9/volume-data-source-validator/0.log" Sep 10 02:48:44.555702 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.555665 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb"] Sep 10 02:48:44.562076 ip-10-0-143-228 kubenswrapper[2570]: W0910 02:48:44.562041 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb99ed8a0_751b_4e2e_a25d_7387de92aa04.slice/crio-9863d3cf0343db9f47b07e053a7702adf66a4ba2a273b2c854ca2c78f5286310 WatchSource:0}: Error finding container 9863d3cf0343db9f47b07e053a7702adf66a4ba2a273b2c854ca2c78f5286310: Status 404 returned error can't find the container with id 9863d3cf0343db9f47b07e053a7702adf66a4ba2a273b2c854ca2c78f5286310 Sep 10 02:48:44.765913 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.765804 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" event={"ID":"b99ed8a0-751b-4e2e-a25d-7387de92aa04","Type":"ContainerStarted","Data":"1c14803c04848f692dd3f0c805cff7204fa0da6ad27a7ecfbe0b0c2968daea0b"} Sep 10 02:48:44.765913 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.765887 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" event={"ID":"b99ed8a0-751b-4e2e-a25d-7387de92aa04","Type":"ContainerStarted","Data":"9863d3cf0343db9f47b07e053a7702adf66a4ba2a273b2c854ca2c78f5286310"} Sep 10 02:48:44.765913 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.765915 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:44.783472 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:44.783422 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" podStartSLOduration=1.783408251 podStartE2EDuration="1.783408251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 02:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 02:48:44.781741404 +0000 UTC m=+934.047837795" watchObservedRunningTime="2026-09-10 02:48:44.783408251 +0000 UTC m=+934.049504641" Sep 10 02:48:45.650573 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:45.650545 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v8xtg_d322117c-a2b5-4598-8c62-31ab1911f3cd/dns/0.log" Sep 10 02:48:45.673105 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:45.673081 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v8xtg_d322117c-a2b5-4598-8c62-31ab1911f3cd/kube-rbac-proxy/0.log" Sep 10 02:48:45.722262 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:45.722229 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gvvrb_55541690-c514-404a-8037-7bb94a66eaf5/dns-node-resolver/0.log" Sep 10 02:48:46.323846 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:46.323805 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w7fk8_1c19c39b-51ec-4ed1-8be8-f99fc93448a5/node-ca/0.log" Sep 10 02:48:47.360594 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:47.360566 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d6b6cf86-z6zzd_97bd02c0-6aeb-49c0-ba0c-3f4521345b6e/kube-auth-proxy/0.log" Sep 10 02:48:47.391048 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:47.391013 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gfpvz_77a55644-04be-4f11-b5a5-ff85419e3bfe/istio-proxy/1.log" Sep 10 02:48:47.391491 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:47.391470 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gfpvz_77a55644-04be-4f11-b5a5-ff85419e3bfe/istio-proxy/0.log" Sep 10 02:48:47.439336 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:47.439307 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-99bf5_d07416bc-6df0-4ddb-acb3-d64eec3d40a9/payload-processing/0.log" Sep 10 02:48:47.472390 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:47.472365 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fb5dd5bbd-khhbd_d22aa7ab-c491-460d-8415-d452a6a64976/router/0.log" Sep 10 02:48:48.124771 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:48.124737 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-whf7d_9d0bb233-da91-40b6-963b-80352b7f96ee/serve-healthcheck-canary/0.log" Sep 10 02:48:48.221981 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:48.221947 2570 scope.go:117] "RemoveContainer" containerID="fbe099a887eb45e9c9aaff79bf24aa663c1ccf6a775be9cb15b8a8875ea17eed" Sep 10 02:48:48.222155 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:48.222106 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-d7mzk_llm(3b89a6d0-864b-4a27-8fa6-cce036e4425d)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-d7mzk" podUID="3b89a6d0-864b-4a27-8fa6-cce036e4425d" Sep 10 02:48:48.581907 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:48.581872 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-bk7gd_92db0478-ff93-4fb6-a696-0885419be325/insights-operator/0.log" Sep 10 02:48:48.583194 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:48.583172 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-bk7gd_92db0478-ff93-4fb6-a696-0885419be325/insights-operator/1.log" Sep 10 02:48:48.605770 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:48.605744 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ssl7_6e07c18b-4023-461c-890c-dd853cda3975/kube-rbac-proxy/0.log" Sep 10 02:48:48.629309 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:48.629285 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ssl7_6e07c18b-4023-461c-890c-dd853cda3975/exporter/0.log" Sep 10 02:48:48.652682 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:48.652656 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ssl7_6e07c18b-4023-461c-890c-dd853cda3975/extractor/0.log" Sep 10 02:48:49.373416 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:49.373376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:48:49.430128 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:49.430074 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmppx_8e1e758b-e536-410e-902e-864dd206dabd/konnectivity-agent/0.log" limit=1024 Sep 10 02:48:50.781272 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:50.781244 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m4dzq/perf-node-gather-daemonset-l9shb" Sep 10 02:48:50.873459 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:50.873429 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pztgq_a94084da-440c-4184-9d87-6447ff838e10/manager/0.log" Sep 10 02:48:50.908946 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:50.908906 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gc4th_49019557-6a1c-4039-ac2f-78844c56798d/manager/0.log" Sep 10 02:48:51.145696 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:51.145582 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647784d6dd-s9mvw_e73accdd-c852-4fd5-b4b1-747844cd7335/manager/0.log" Sep 10 02:48:51.166640 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:51.166614 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-647784d6dd-s9mvw_e73accdd-c852-4fd5-b4b1-747844cd7335/copy-manifests/0.log" Sep 10 02:48:54.098791 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:54.098764 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d6b6cf86-z6zzd_97bd02c0-6aeb-49c0-ba0c-3f4521345b6e/kube-auth-proxy/0.log" Sep 10 02:48:54.121534 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:54.121504 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gfpvz_77a55644-04be-4f11-b5a5-ff85419e3bfe/istio-proxy/1.log" Sep 10 02:48:54.121988 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:54.121969 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gfpvz_77a55644-04be-4f11-b5a5-ff85419e3bfe/istio-proxy/0.log" Sep 10 02:48:54.155617 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:54.155533 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-99bf5_d07416bc-6df0-4ddb-acb3-d64eec3d40a9/payload-processing/0.log" Sep 10 02:48:54.177668 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:54.177647 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fb5dd5bbd-khhbd_d22aa7ab-c491-460d-8415-d452a6a64976/router/0.log" Sep 10 02:48:55.313467 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:55.313441 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74d6b6cf86-z6zzd_97bd02c0-6aeb-49c0-ba0c-3f4521345b6e/kube-auth-proxy/0.log" Sep 10 02:48:55.336451 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:55.336426 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gfpvz_77a55644-04be-4f11-b5a5-ff85419e3bfe/istio-proxy/1.log" Sep 10 02:48:55.336775 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:55.336760 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-gfpvz_77a55644-04be-4f11-b5a5-ff85419e3bfe/istio-proxy/0.log" Sep 10 02:48:55.371374 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:55.371347 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-99bf5_d07416bc-6df0-4ddb-acb3-d64eec3d40a9/payload-processing/0.log" Sep 10 02:48:55.389849 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:55.389806 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fb5dd5bbd-khhbd_d22aa7ab-c491-460d-8415-d452a6a64976/router/0.log" Sep 10 02:48:56.221817 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:56.221786 2570 scope.go:117] "RemoveContainer" containerID="30ddaed8a7bf126d7a39dbae510bd189baa55bb4fe1af4240bb84748c9b1197b" Sep 10 02:48:56.222012 ip-10-0-143-228 kubenswrapper[2570]: E0910 02:48:56.221995 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47_llm(78e0a214-7b96-4a60-97c1-9e3619b9c00a)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-d7s47" podUID="78e0a214-7b96-4a60-97c1-9e3619b9c00a" Sep 10 02:48:57.790298 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:57.790266 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-zknhx_efa04a19-d382-4e5e-9cdb-a48b40b57220/migrator/0.log" Sep 10 02:48:57.818636 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:57.818599 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-zknhx_efa04a19-d382-4e5e-9cdb-a48b40b57220/graceful-termination/0.log" Sep 10 02:48:58.190259 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:58.190230 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-szhxg_24a99e6f-fe48-48fc-a340-09ae9d1728fc/kube-storage-version-migrator-operator/1.log" Sep 10 02:48:58.191157 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:58.191139 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-szhxg_24a99e6f-fe48-48fc-a340-09ae9d1728fc/kube-storage-version-migrator-operator/0.log" Sep 10 02:48:59.262814 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.262782 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8xc2b_14a6ac2d-65fb-4070-b0f0-979edbcf7aa9/kube-multus/0.log" Sep 10 02:48:59.446137 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.446110 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-87cr5_85a8b7bf-b25c-49fc-8889-80653c50b02b/kube-multus-additional-cni-plugins/0.log" Sep 10 02:48:59.467450 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.467426 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-87cr5_85a8b7bf-b25c-49fc-8889-80653c50b02b/egress-router-binary-copy/0.log" Sep 10 02:48:59.488693 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.488640 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-87cr5_85a8b7bf-b25c-49fc-8889-80653c50b02b/cni-plugins/0.log" Sep 10 02:48:59.515577 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.515501 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-87cr5_85a8b7bf-b25c-49fc-8889-80653c50b02b/bond-cni-plugin/0.log" Sep 10 02:48:59.535898 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.535871 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-87cr5_85a8b7bf-b25c-49fc-8889-80653c50b02b/routeoverride-cni/0.log" Sep 10 02:48:59.559317 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.559290 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-87cr5_85a8b7bf-b25c-49fc-8889-80653c50b02b/whereabouts-cni-bincopy/0.log" Sep 10 02:48:59.581122 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.581085 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-87cr5_85a8b7bf-b25c-49fc-8889-80653c50b02b/whereabouts-cni/0.log" Sep 10 02:48:59.866792 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.866716 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6hhxs_253d3186-27d1-4b7e-b48a-303871be8896/network-metrics-daemon/0.log" Sep 10 02:48:59.890792 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:48:59.890762 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6hhxs_253d3186-27d1-4b7e-b48a-303871be8896/kube-rbac-proxy/0.log" Sep 10 02:49:01.113392 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.113359 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/ovn-controller/0.log" Sep 10 02:49:01.132415 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.132390 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/ovn-acl-logging/0.log" Sep 10 02:49:01.158603 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.158573 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/kube-rbac-proxy-node/0.log" Sep 10 02:49:01.181610 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.181579 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 02:49:01.202066 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.202036 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/northd/0.log" Sep 10 02:49:01.222886 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.222859 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/nbdb/0.log" Sep 10 02:49:01.244260 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.244233 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/sbdb/0.log" Sep 10 02:49:01.285116 ip-10-0-143-228 kubenswrapper[2570]: I0910 02:49:01.285090 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs6vw_ca08812a-7a23-4716-8d22-e116d09a39a4/ovnkube-controller/0.log"