Jul 23 23:11:10.731236 ip-10-0-135-101 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 23:11:10.731396 ip-10-0-135-101 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 23:11:10.731411 ip-10-0-135-101 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 23:11:10.732490 ip-10-0-135-101 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 23:11:20.954177 ip-10-0-135-101 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 23:11:20.954192 ip-10-0-135-101 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2a7c43a90125446f96fbd49de30508b5 -- Jul 23 23:12:55.090344 ip-10-0-135-101 systemd[1]: Starting Kubernetes Kubelet... Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.541851 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548595 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548608 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548614 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548618 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548621 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548626 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548629 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 23:12:55.551485 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548632 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548634 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548639 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548642 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548645 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548653 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548656 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548662 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548664 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548667 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548670 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548675 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548677 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548680 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548685 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548687 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548690 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548692 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548697 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548700 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 23:12:55.558114 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548703 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548708 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548711 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548714 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548716 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548722 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548724 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548729 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548735 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548737 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548740 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548745 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548750 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548754 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548762 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548766 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548783 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548787 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548794 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 23:12:55.559922 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548802 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548806 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548811 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548817 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548820 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548826 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548830 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548870 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548880 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548886 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548890 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548894 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548900 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548903 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548910 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548913 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548916 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548922 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548925 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548929 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 23:12:55.562253 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548934 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548937 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548941 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548946 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548949 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548957 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548962 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548970 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548973 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548976 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548978 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548981 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548984 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548990 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548992 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.548995 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549000 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549003 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549007 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 23:12:55.573828 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549013 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549016 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549018 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549023 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549026 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549028 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549031 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549036 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549038 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549041 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549044 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549049 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549051 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549054 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549912 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549922 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549925 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549927 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549933 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549937 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 23:12:55.576107 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549942 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549945 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549947 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549950 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549957 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549959 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549962 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549966 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549969 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549972 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549975 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549980 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549982 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549985 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549990 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549993 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549996 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.549999 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550003 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 23:12:55.578070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550006 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550009 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550014 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550018 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550022 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550030 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550033 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550036 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550038 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550041 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550044 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550051 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550053 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550057 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550061 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550064 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550067 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550071 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550090 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 23:12:55.580618 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550096 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550103 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550108 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550113 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550118 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550122 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550126 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550130 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550134 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550138 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550142 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550146 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550152 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550157 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550162 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550167 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550171 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550175 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550179 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550184 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 23:12:55.593815 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550211 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550217 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550222 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550227 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550232 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550236 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550241 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550247 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550251 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550262 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550266 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550270 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550274 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550281 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550284 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550288 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550292 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550295 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550298 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550305 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 23:12:55.596128 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550308 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550313 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550315 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550318 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550324 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550327 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550331 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550336 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550340 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550344 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550348 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550353 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550356 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550361 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550364 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550798 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550803 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550809 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550811 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550814 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 23:12:55.597988 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550819 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550822 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550825 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550827 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550832 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550834 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550837 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550840 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550844 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550847 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550850 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550852 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550857 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550859 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550862 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550864 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550871 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550874 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550881 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550884 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 23:12:55.600466 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550886 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550891 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550894 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550897 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550900 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550904 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550908 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550914 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550917 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550920 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550922 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550925 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550928 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550930 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550937 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550939 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550942 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550947 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550949 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550952 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 23:12:55.613762 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550954 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550959 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550962 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550964 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550967 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550971 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550974 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550976 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550982 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550987 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550994 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.550998 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551000 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551006 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551009 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551011 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551016 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551019 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551024 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 23:12:55.616216 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551029 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551032 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551035 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551040 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551043 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551046 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551051 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551054 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551057 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551062 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551065 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551068 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551089 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551093 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551098 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551101 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551104 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551108 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551112 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 23:12:55.618070 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551118 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551120 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551125 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551128 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551132 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551134 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551139 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551141 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551144 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551146 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551151 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551154 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551156 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551159 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551164 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.551167 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551249 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551257 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551263 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551269 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551274 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 23:12:55.620657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551279 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551283 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551287 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551292 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551295 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551299 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551304 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551307 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551310 2568 flags.go:64] FLAG: --cgroup-root="" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551315 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551320 2568 flags.go:64] FLAG: --client-ca-file="" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551326 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551329 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551336 2568 flags.go:64] FLAG: --cluster-domain="" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551338 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551343 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551348 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551351 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551355 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551359 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551362 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551366 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551371 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551374 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 23:12:55.714639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551376 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 23:12:55.625934 ip-10-0-135-101 systemd[1]: Started Kubernetes Kubelet. Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551381 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551386 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551393 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551397 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551401 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551405 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551409 2568 flags.go:64] FLAG: --enable-server="true" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551415 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551421 2568 flags.go:64] FLAG: --event-burst="100" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551425 2568 flags.go:64] FLAG: --event-qps="50" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551429 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551433 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551438 2568 flags.go:64] FLAG: --eviction-hard="" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551451 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551455 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551460 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551464 2568 flags.go:64] FLAG: --eviction-soft="" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551469 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551474 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551478 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551484 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551489 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551494 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551499 2568 flags.go:64] FLAG: --feature-gates="" Jul 23 23:12:55.862335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551505 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551510 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551514 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551519 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551523 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551527 2568 flags.go:64] FLAG: --help="false" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551531 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-135-101.ec2.internal" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551535 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551539 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551544 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551548 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551553 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551557 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551561 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551565 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551569 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551574 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551578 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551582 2568 flags.go:64] FLAG: --kube-reserved="" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551586 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551590 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551595 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551599 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551603 2568 flags.go:64] FLAG: --lock-file="" Jul 23 23:12:55.868819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551606 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551611 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551615 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551622 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551627 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551632 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551637 2568 flags.go:64] FLAG: --logging-format="text" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551642 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551646 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551651 2568 flags.go:64] FLAG: --manifest-url="" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551655 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551661 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551665 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551670 2568 flags.go:64] FLAG: --max-pods="110" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551674 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551679 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551683 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551687 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551691 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551695 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551699 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551709 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551713 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551717 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 23:12:55.869838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551721 2568 flags.go:64] FLAG: --pod-cidr="" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551726 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551734 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551738 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551743 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551748 2568 flags.go:64] FLAG: --port="10250" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551753 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551757 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a1c0b548acae16ac" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551762 2568 flags.go:64] FLAG: --qos-reserved="" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551766 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551770 2568 flags.go:64] FLAG: --register-node="true" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551774 2568 flags.go:64] FLAG: --register-with-taints="" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551778 2568 flags.go:64] FLAG: --registry-burst="10" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551783 2568 flags.go:64] FLAG: --registry-qps="5" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551787 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551791 2568 flags.go:64] FLAG: --reserved-memory="" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551796 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551800 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551804 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551808 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551812 2568 flags.go:64] FLAG: --runonce="false" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551816 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551820 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551824 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551828 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 23:12:55.870685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551832 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551836 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551841 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551845 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551849 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551854 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551859 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551864 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551869 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551881 2568 flags.go:64] FLAG: --system-cgroups="" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551886 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551897 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551903 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551907 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551915 2568 flags.go:64] FLAG: --tls-min-version="" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551920 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551924 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551929 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551934 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551939 2568 flags.go:64] FLAG: --v="2" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551946 2568 flags.go:64] FLAG: --version="false" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551952 2568 flags.go:64] FLAG: --vmodule="" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551959 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.551964 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552161 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 23:12:55.871639 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552170 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552175 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552180 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552184 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552188 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552192 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552199 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552203 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552207 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552211 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552216 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552220 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552225 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552229 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552233 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552237 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552241 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552247 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552253 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 23:12:55.872431 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552257 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552262 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552266 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552270 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552274 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552278 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552283 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552287 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552291 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552296 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552300 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552304 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552308 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552312 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552318 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552325 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552330 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552334 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552338 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552345 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 23:12:55.873069 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552349 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552353 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552357 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552361 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552365 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552369 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552373 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552378 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552382 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552386 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552390 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552397 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552401 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552405 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552410 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552414 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552418 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552422 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552426 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 23:12:55.873704 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552430 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552434 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552439 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552443 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552448 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552452 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552456 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552460 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552464 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552469 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552473 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552476 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552483 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552490 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552495 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552500 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552504 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552509 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552513 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552524 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 23:12:55.874332 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552528 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552532 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552537 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552541 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552547 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552552 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552559 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552564 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552569 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552573 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552578 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552582 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552586 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552590 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552594 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552599 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552603 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552608 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552612 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 23:12:55.874984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.552616 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.553614 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.561070 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.561100 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.561114 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.561123 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561186 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561191 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561194 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561198 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561201 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561204 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561206 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561209 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561212 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 23:12:55.875655 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561215 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561217 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561220 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561223 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561225 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561228 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561230 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561233 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561235 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561238 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561241 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561243 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561247 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561251 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561255 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561260 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561263 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561266 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561268 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561271 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 23:12:55.876156 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561279 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561282 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561285 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561287 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561290 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561292 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561295 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561297 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561300 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561302 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561305 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561307 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561310 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561312 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561316 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561319 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561322 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561324 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561327 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561330 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 23:12:55.876869 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561333 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561335 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561338 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561340 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561343 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561345 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561348 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561352 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561355 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561357 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561360 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561363 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561365 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561373 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561376 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561378 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561381 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561383 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561386 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561389 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 23:12:55.877447 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561391 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561394 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561402 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561405 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561408 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561412 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561414 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561417 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561419 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561422 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561424 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561427 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561430 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561432 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561435 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561438 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561442 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561445 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561448 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 23:12:55.878034 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561453 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561456 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561459 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561461 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561464 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561467 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561470 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561480 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561484 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561487 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561491 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.561496 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561643 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561648 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561652 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 23:12:55.878581 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561654 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561657 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561660 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561662 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561665 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561668 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561670 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561673 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561675 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561677 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561680 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561683 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561685 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561687 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561690 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561692 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561695 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561697 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561701 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561703 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 23:12:55.879040 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561706 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561708 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561711 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561714 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561716 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561724 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561727 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561729 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561732 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561736 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561738 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561741 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561743 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561746 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561748 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561751 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561753 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561756 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561758 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561761 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 23:12:55.879599 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561765 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561769 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561772 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561775 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561778 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561781 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561783 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561786 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561789 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561791 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561795 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561798 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561800 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561803 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561805 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561808 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561810 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561813 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561821 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 23:12:55.880237 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561824 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561826 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561830 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561833 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561835 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561838 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561840 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561843 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561845 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561848 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561850 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561853 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561855 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561858 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561860 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561863 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561865 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561868 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561870 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 23:12:55.880800 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561872 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561875 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561877 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561881 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561887 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561890 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561893 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561896 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561898 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561901 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561903 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561906 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561909 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561917 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561921 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561925 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561928 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 23:12:55.979578 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:55.561930 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.561935 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.562058 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.565303 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.567010 2568 server.go:1019] "Starting client certificate rotation" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.567125 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.567161 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.570000 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.573222 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.588653 2568 log.go:25] "Validated CRI v1 runtime API" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.592309 2568 log.go:25] "Validated CRI v1 image API" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.594237 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 23:12:55.982182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.598609 2568 fs.go:135] Filesystem UUIDs: map[0bd51ec3-61db-4abd-a6b4-5c5fa79d9d25:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a21cd14b-cebc-4f2d-9fab-29471887f8a3:/dev/nvme0n1p3] Jul 23 23:12:55.982629 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.598622 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 23:12:55.982629 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.603672 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.604819 2568 manager.go:216] Machine: {Timestamp:2026-07-23 23:12:55.60336713 +0000 UTC m=+0.383978334 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096401 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b19f0d30d6bacf2fd4b8e57fe5334 SystemUUID:ec2b19f0-d30d-6bac-f2fd-4b8e57fe5334 BootID:2a7c43a9-0125-446f-96fb-d49de30508b5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:08:45:37:52:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:08:45:37:52:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:73:93:45:ee:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.604921 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.604993 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.607607 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.607630 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.608509 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.608525 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.608532 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.608546 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.609768 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.612699 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.612820 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.612845 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.612856 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.612868 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.612879 2568 kubelet.go:398] "Adding apiserver pod source" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.612890 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 23:12:55.983583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.613224 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.613247 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.615600 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.619372 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.619412 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621660 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621678 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621688 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621725 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621737 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621744 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621753 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621759 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621767 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621778 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621795 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621801 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621818 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.621824 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.623034 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.623138 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.625210 2568 server.go:1268] "Started kubelet" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.625296 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.625269 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.625463 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.627200 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.627515 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.630669 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.633942 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.633805 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-101.ec2.internal.18c50e241f1276ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-101.ec2.internal,UID:ip-10-0-135-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-101.ec2.internal,},FirstTimestamp:2026-07-23 23:12:55.625184974 +0000 UTC m=+0.405796178,LastTimestamp:2026-07-23 23:12:55.625184974 +0000 UTC m=+0.405796178,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-101.ec2.internal,}" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.637379 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.637386 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.637460 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638171 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638173 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638196 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638204 2568 factory.go:55] Registering systemd factory Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638218 2568 factory.go:223] Registration of the systemd container factory successfully Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638318 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638330 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638410 2568 factory.go:153] Registering CRI-O factory Jul 23 23:12:56.107406 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638423 2568 factory.go:223] Registration of the crio container factory successfully Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638471 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638497 2568 factory.go:103] Registering Raw factory Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638513 2568 manager.go:1196] Started watching for new ooms in manager Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.638763 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.638848 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.638985 2568 manager.go:349] Starting recovery of all containers Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.651837 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cvkkg" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.652357 2568 manager.go:354] Recovery completed Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.656064 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.657416 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cvkkg" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.657890 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.657914 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.657925 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.658435 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.658446 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.658453 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.658464 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.659020 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.659042 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.659050 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.662145 2568 policy_none.go:49] "None policy: Start" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.662162 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.662174 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.664118 2568 policy_none.go:47] "Start" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.738166 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.740748 2568 manager.go:337] "Starting Device Plugin manager" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.740800 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.740810 2568 server.go:87] "Starting device plugin registration server" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.741094 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.741113 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.741177 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.741297 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.741308 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.741759 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.741801 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.756212 2568 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2603 into the system container \"/system.slice\": " Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.841274 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.842201 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.842230 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.842239 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.842263 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.845584 2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-101.ec2.internal\" not found" node="ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.108789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.852136 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.852155 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-101.ec2.internal\": node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.876459 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.905561 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.906904 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.906937 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.906958 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.906989 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:55.909409 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:55.976786 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.007908 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal"] Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.007981 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.008871 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.008899 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.008911 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.011309 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.011436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.011463 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.011985 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.011992 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.012012 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.012025 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.012014 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.012123 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.014272 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.014292 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.014906 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientMemory" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.014933 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.014961 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeHasSufficientPID" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.046621 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5914545796d27e5f7bd262f1db2d1af6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"5914545796d27e5f7bd262f1db2d1af6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.046648 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5914545796d27e5f7bd262f1db2d1af6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"5914545796d27e5f7bd262f1db2d1af6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.046667 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed436c2c06eb421aaa817d4e8a2cf780-config\") pod \"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal\" (UID: \"ed436c2c06eb421aaa817d4e8a2cf780\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.046781 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-101.ec2.internal\" not found" node="ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.051129 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-101.ec2.internal\" not found" node="ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.110024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.078414 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.146996 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5914545796d27e5f7bd262f1db2d1af6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"5914545796d27e5f7bd262f1db2d1af6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.147017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5914545796d27e5f7bd262f1db2d1af6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"5914545796d27e5f7bd262f1db2d1af6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.147034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed436c2c06eb421aaa817d4e8a2cf780-config\") pod \"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal\" (UID: \"ed436c2c06eb421aaa817d4e8a2cf780\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.147058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed436c2c06eb421aaa817d4e8a2cf780-config\") pod \"kube-apiserver-proxy-ip-10-0-135-101.ec2.internal\" (UID: \"ed436c2c06eb421aaa817d4e8a2cf780\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.147091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5914545796d27e5f7bd262f1db2d1af6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"5914545796d27e5f7bd262f1db2d1af6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.147116 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5914545796d27e5f7bd262f1db2d1af6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal\" (UID: \"5914545796d27e5f7bd262f1db2d1af6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.237997 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.179101 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.363989 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.279586 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.363989 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.351118 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.363989 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.354240 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 23 23:12:56.510687 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.379706 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.510687 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.460501 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 23:12:56.510687 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.480769 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:56.650130 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.566291 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 23:12:56.650130 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.566394 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 23:12:56.650130 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.566429 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 23:12:56.650130 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.585526 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:58.046532 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.663952 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 23:07:56 +0000 UTC" deadline="2028-03-08 09:11:02.356569466 +0000 UTC" Jul 23 23:12:58.046532 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:56.663975 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14241h58m5.692596784s" Jul 23 23:12:58.046532 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:56.686098 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:58.046532 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:58.038941 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:58.165727 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:58.139453 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-101.ec2.internal\" not found" Jul 23 23:12:58.275756 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.184770 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 23:12:58.275756 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.190219 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" Jul 23 23:12:58.275756 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.201868 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 23:12:58.275756 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.202749 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" Jul 23 23:12:58.275756 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.210761 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 23:12:58.415392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.299944 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 23:12:58.415392 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:58.320176 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded436c2c06eb421aaa817d4e8a2cf780.slice/crio-2ac7edf0205d8bfc097ff8afd3b6af63a64fd9753abe212cb9b1a1a085668b15 WatchSource:0}: Error finding container 2ac7edf0205d8bfc097ff8afd3b6af63a64fd9753abe212cb9b1a1a085668b15: Status 404 returned error can't find the container with id 2ac7edf0205d8bfc097ff8afd3b6af63a64fd9753abe212cb9b1a1a085668b15 Jul 23 23:12:58.415392 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:58.320356 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5914545796d27e5f7bd262f1db2d1af6.slice/crio-75053c5174db9f4972208ca6b138eb2d4472e55dc5cfdf36a774d919e1a293ae WatchSource:0}: Error finding container 75053c5174db9f4972208ca6b138eb2d4472e55dc5cfdf36a774d919e1a293ae: Status 404 returned error can't find the container with id 75053c5174db9f4972208ca6b138eb2d4472e55dc5cfdf36a774d919e1a293ae Jul 23 23:12:58.415392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.324736 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 23:12:58.767280 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.766262 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 23:12:58.875981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.875753 2568 apiserver.go:52] "Watching apiserver" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.881578 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.883106 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-wz6vs","openshift-image-registry/node-ca-nlggj","openshift-network-diagnostics/network-check-target-9bxbv","openshift-ovn-kubernetes/ovnkube-node-zb8ln","kube-system/global-pull-secret-syncer-hc8bv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal","openshift-multus/multus-5vxfr","openshift-multus/multus-additional-cni-plugins-stvz9","openshift-multus/network-metrics-daemon-xdbwg","openshift-network-operator/iptables-alerter-r6774","kube-system/konnectivity-agent-7n678","kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal","openshift-cluster-node-tuning-operator/tuned-j2s62"] Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.885825 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:58.885913 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.887899 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.890083 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:58.890180 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.890237 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.892522 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.893314 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.893338 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.893647 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.893679 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zbvlq\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.895338 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:58.895449 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.895989 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.896494 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.896708 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.897033 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.897388 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.897393 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7799t\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.896497 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.900446 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.902883 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.902911 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.902981 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.902996 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kgfkl\"" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.903223 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:58.938427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.904948 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.904964 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905334 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905362 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905520 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905532 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905612 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5g97f\"" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905694 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovnkube-script-lib\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905716 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905737 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-systemd-units\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905756 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-etc-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905775 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-node-log\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.905961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttmj\" (UniqueName: \"kubernetes.io/projected/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-kube-api-access-tttmj\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906036 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-systemd\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906059 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-ovn\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-kubelet\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906126 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-slash\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.940241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-run-netns\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-iptables-alerter-script\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-host-slash\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz89z\" (UniqueName: \"kubernetes.io/projected/afe83ceb-ece6-4972-b6ef-cf1671100c7b-kube-api-access-rz89z\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgd2\" (UniqueName: \"kubernetes.io/projected/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-kube-api-access-pvgd2\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906481 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906502 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-var-lib-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-env-overrides\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906560 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-log-socket\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-cni-bin\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-cni-netd\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.906654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovnkube-config\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.907673 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d6wdj\"" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.907682 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.907703 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.908011 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.909665 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t5cvp\"" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.909681 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 23:12:58.943122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.909975 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.910037 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.910468 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.914753 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dbqpq\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.914775 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.914783 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.915192 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5vxfr" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.918510 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.918647 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2b9vc\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.920189 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.921884 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ttt7t\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.922025 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 23:12:58.944211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.922138 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 23:12:59.006863 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.959294 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h5wn6" Jul 23 23:12:59.006863 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.973202 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h5wn6" Jul 23 23:12:59.006863 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:58.991897 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007589 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-cnibin\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007663 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mfk\" (UniqueName: \"kubernetes.io/projected/03fd2de7-bc46-4700-aa22-e765108fe445-kube-api-access-27mfk\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-iptables-alerter-script\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007731 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-host-slash\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4qc\" (UniqueName: \"kubernetes.io/projected/4be791d5-2577-4234-9289-6c870b08b7f8-kube-api-access-tx4qc\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007822 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgd2\" (UniqueName: \"kubernetes.io/projected/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-kube-api-access-pvgd2\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007891 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-var-lib-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxsb\" (UniqueName: \"kubernetes.io/projected/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-kube-api-access-9vxsb\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-cni-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.007981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-k8s-cni-cncf-io\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.008142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-iptables-alerter-script\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e93b0ed8-c722-414d-8752-4dac337b6739-agent-certs\") pod \"konnectivity-agent-7n678\" (UID: \"e93b0ed8-c722-414d-8752-4dac337b6739\") " pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-socket-dir-parent\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-cni-bin\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008598 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-log-socket\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-cni-bin\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008636 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-host-slash\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008752 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-log-socket\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008806 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-cni-bin\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008844 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-var-lib-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008643 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.008988 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63158caf-bd4e-4285-9544-e5abb2424302-hosts-file\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63158caf-bd4e-4285-9544-e5abb2424302-tmp-dir\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009048 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-multus-certs\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009081 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-cni-netd\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009118 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovnkube-config\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovnkube-script-lib\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.011371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6659ad44-c467-47ce-b25e-33dbcc41f0ba-tmp\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-netns\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-hostroot\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009148 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-cni-netd\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03fd2de7-bc46-4700-aa22-e765108fe445-multus-daemon-config\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-systemd-units\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009432 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-etc-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009462 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-systemd-units\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009527 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-host\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-etc-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009589 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-etc-kubernetes\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-systemd\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-systemd\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009678 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovnkube-config\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.012437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-ovn\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d90782c3-1c42-4715-804d-63e1d26969f2-dbus\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-cnibin\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009724 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-ovn\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009816 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovnkube-script-lib\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009836 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-kubelet\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-slash\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009878 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-system-cni-dir\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009899 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009900 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-kubelet\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-slash\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009942 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-modprobe-d\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.009969 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-sys\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010000 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxvd\" (UniqueName: \"kubernetes.io/projected/6659ad44-c467-47ce-b25e-33dbcc41f0ba-kube-api-access-llxvd\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz89z\" (UniqueName: \"kubernetes.io/projected/afe83ceb-ece6-4972-b6ef-cf1671100c7b-kube-api-access-rz89z\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.013920 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be791d5-2577-4234-9289-6c870b08b7f8-serviceca\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010095 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-var-lib-kubelet\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-tuned\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010162 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-env-overrides\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysctl-conf\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010231 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-os-release\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d90782c3-1c42-4715-804d-63e1d26969f2-kubelet-config\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-os-release\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010297 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be791d5-2577-4234-9289-6c870b08b7f8-host\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010316 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-device-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-kubernetes\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010356 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6xp\" (UniqueName: \"kubernetes.io/projected/63158caf-bd4e-4285-9544-e5abb2424302-kube-api-access-9r6xp\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkhz\" (UniqueName: \"kubernetes.io/projected/47c58c88-b124-4e13-8596-79c1031522fa-kube-api-access-pfkhz\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.016045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010437 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010456 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-lib-modules\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-node-log\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010539 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysctl-d\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-run\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010573 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/afe83ceb-ece6-4972-b6ef-cf1671100c7b-env-overrides\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-system-cni-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03fd2de7-bc46-4700-aa22-e765108fe445-cni-binary-copy\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-conf-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010683 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-run-openvswitch\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010703 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-node-log\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010700 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysconfig\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010758 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-systemd\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttmj\" (UniqueName: \"kubernetes.io/projected/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-kube-api-access-tttmj\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.010794 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:12:59.017122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010822 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-cni-multus\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.010898 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs podName:7933aa4d-a215-4a78-bcfd-ec12e3d6cf92 nodeName:}" failed. No retries permitted until 2026-07-23 23:12:59.510878741 +0000 UTC m=+3.039155906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs") pod "network-metrics-daemon-xdbwg" (UID: "7933aa4d-a215-4a78-bcfd-ec12e3d6cf92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.010981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-kubelet\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.011005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e93b0ed8-c722-414d-8752-4dac337b6739-konnectivity-ca\") pod \"konnectivity-agent-7n678\" (UID: \"e93b0ed8-c722-414d-8752-4dac337b6739\") " pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.011028 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-run-netns\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.011058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afe83ceb-ece6-4972-b6ef-cf1671100c7b-host-run-netns\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.012896 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/afe83ceb-ece6-4972-b6ef-cf1671100c7b-ovn-node-metrics-cert\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.016444 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.016466 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.016479 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jfvrc for pod openshift-network-diagnostics/network-check-target-9bxbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:12:59.017908 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.016857 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc podName:620d6fc7-90b1-462c-8644-92d565a7d9e7 nodeName:}" failed. No retries permitted until 2026-07-23 23:12:59.516825142 +0000 UTC m=+3.045102312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfvrc" (UniqueName: "kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc") pod "network-check-target-9bxbv" (UID: "620d6fc7-90b1-462c-8644-92d565a7d9e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:12:59.024305 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.024240 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz89z\" (UniqueName: \"kubernetes.io/projected/afe83ceb-ece6-4972-b6ef-cf1671100c7b-kube-api-access-rz89z\") pod \"ovnkube-node-zb8ln\" (UID: \"afe83ceb-ece6-4972-b6ef-cf1671100c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.037173 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.028539 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgd2\" (UniqueName: \"kubernetes.io/projected/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-kube-api-access-pvgd2\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:12:59.037173 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.028678 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttmj\" (UniqueName: \"kubernetes.io/projected/e88142f1-c2db-42c8-bc93-4f5b09b5d8b4-kube-api-access-tttmj\") pod \"iptables-alerter-r6774\" (UID: \"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4\") " pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:59.062186 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.061866 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6774" Jul 23 23:12:59.066025 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.065898 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:12:59.111454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxsb\" (UniqueName: \"kubernetes.io/projected/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-kube-api-access-9vxsb\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.111454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-cni-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-k8s-cni-cncf-io\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e93b0ed8-c722-414d-8752-4dac337b6739-agent-certs\") pod \"konnectivity-agent-7n678\" (UID: \"e93b0ed8-c722-414d-8752-4dac337b6739\") " pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:59.111454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111433 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-socket-dir-parent\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-cni-bin\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111481 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63158caf-bd4e-4285-9544-e5abb2424302-hosts-file\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63158caf-bd4e-4285-9544-e5abb2424302-tmp-dir\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-multus-certs\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6659ad44-c467-47ce-b25e-33dbcc41f0ba-tmp\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-netns\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-hostroot\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03fd2de7-bc46-4700-aa22-e765108fe445-multus-daemon-config\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.111737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111729 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-socket-dir-parent\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-host\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111775 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-etc-kubernetes\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d90782c3-1c42-4715-804d-63e1d26969f2-dbus\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-cnibin\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111917 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-system-cni-dir\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.111966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-modprobe-d\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-sys\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxvd\" (UniqueName: \"kubernetes.io/projected/6659ad44-c467-47ce-b25e-33dbcc41f0ba-kube-api-access-llxvd\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be791d5-2577-4234-9289-6c870b08b7f8-serviceca\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112180 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-k8s-cni-cncf-io\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112184 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-var-lib-kubelet\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.112236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-tuned\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112247 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-var-lib-kubelet\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysctl-conf\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-os-release\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d90782c3-1c42-4715-804d-63e1d26969f2-kubelet-config\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112336 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-os-release\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be791d5-2577-4234-9289-6c870b08b7f8-host\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112366 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-device-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-kubernetes\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6xp\" (UniqueName: \"kubernetes.io/projected/63158caf-bd4e-4285-9544-e5abb2424302-kube-api-access-9r6xp\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112435 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-socket-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112444 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkhz\" (UniqueName: \"kubernetes.io/projected/47c58c88-b124-4e13-8596-79c1031522fa-kube-api-access-pfkhz\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-lib-modules\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112505 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/63158caf-bd4e-4285-9544-e5abb2424302-hosts-file\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-cni-bin\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.115899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112538 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d90782c3-1c42-4715-804d-63e1d26969f2-dbus\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysctl-d\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-run\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112604 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-system-cni-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112629 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03fd2de7-bc46-4700-aa22-e765108fe445-cni-binary-copy\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112633 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-system-cni-dir\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112643 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-conf-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysconfig\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112690 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4be791d5-2577-4234-9289-6c870b08b7f8-host\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112700 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-registration-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-systemd\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112754 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-cni-multus\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112810 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-kubelet\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e93b0ed8-c722-414d-8752-4dac337b6739-konnectivity-ca\") pod \"konnectivity-agent-7n678\" (UID: \"e93b0ed8-c722-414d-8752-4dac337b6739\") " pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112917 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-lib-modules\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.116572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-cnibin\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mfk\" (UniqueName: \"kubernetes.io/projected/03fd2de7-bc46-4700-aa22-e765108fe445-kube-api-access-27mfk\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112962 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-sys\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112973 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-run\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113009 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4qc\" (UniqueName: \"kubernetes.io/projected/4be791d5-2577-4234-9289-6c870b08b7f8-kube-api-access-tx4qc\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysctl-d\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-cnibin\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113126 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63158caf-bd4e-4285-9544-e5abb2424302-tmp-dir\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113158 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d90782c3-1c42-4715-804d-63e1d26969f2-kubelet-config\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113196 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-multus-certs\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-system-cni-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113218 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-modprobe-d\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-device-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-systemd\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-os-release\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113339 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-kubernetes\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-cnibin\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-conf-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.117583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113596 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4be791d5-2577-4234-9289-6c870b08b7f8-serviceca\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113676 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-sys-fs\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113708 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112144 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-multus-cni-dir\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113766 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-os-release\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113787 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-etc-kubernetes\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113794 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-run-netns\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113789 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-hostroot\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.113849 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.113882 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.113943 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret podName:d90782c3-1c42-4715-804d-63e1d26969f2 nodeName:}" failed. No retries permitted until 2026-07-23 23:12:59.613924776 +0000 UTC m=+3.142201931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret") pod "global-pull-secret-syncer-hc8bv" (UID: "d90782c3-1c42-4715-804d-63e1d26969f2") : object "kube-system"/"original-pull-secret" not registered Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.114040 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-host\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.114130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-cni-multus\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.114362 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47c58c88-b124-4e13-8596-79c1031522fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.114602 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.114627 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03fd2de7-bc46-4700-aa22-e765108fe445-host-var-lib-kubelet\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.112811 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysctl-conf\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.118956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.114657 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-sysconfig\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.119869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.115040 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03fd2de7-bc46-4700-aa22-e765108fe445-cni-binary-copy\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.119869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.115094 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47c58c88-b124-4e13-8596-79c1031522fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.119869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.115188 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e93b0ed8-c722-414d-8752-4dac337b6739-konnectivity-ca\") pod \"konnectivity-agent-7n678\" (UID: \"e93b0ed8-c722-414d-8752-4dac337b6739\") " pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:59.119869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.115657 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03fd2de7-bc46-4700-aa22-e765108fe445-multus-daemon-config\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.119869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.116835 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6659ad44-c467-47ce-b25e-33dbcc41f0ba-tmp\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.119869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.117529 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e93b0ed8-c722-414d-8752-4dac337b6739-agent-certs\") pod \"konnectivity-agent-7n678\" (UID: \"e93b0ed8-c722-414d-8752-4dac337b6739\") " pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:59.120701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.120673 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6659ad44-c467-47ce-b25e-33dbcc41f0ba-etc-tuned\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.122174 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.122134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxsb\" (UniqueName: \"kubernetes.io/projected/c6e4d568-8bcb-43f9-87ea-6ee61e318bcf-kube-api-access-9vxsb\") pod \"aws-ebs-csi-driver-node-xzn5n\" (UID: \"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.124424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.124335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkhz\" (UniqueName: \"kubernetes.io/projected/47c58c88-b124-4e13-8596-79c1031522fa-kube-api-access-pfkhz\") pod \"multus-additional-cni-plugins-stvz9\" (UID: \"47c58c88-b124-4e13-8596-79c1031522fa\") " pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.126390 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.126368 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxvd\" (UniqueName: \"kubernetes.io/projected/6659ad44-c467-47ce-b25e-33dbcc41f0ba-kube-api-access-llxvd\") pod \"tuned-j2s62\" (UID: \"6659ad44-c467-47ce-b25e-33dbcc41f0ba\") " pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.126750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.126695 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4qc\" (UniqueName: \"kubernetes.io/projected/4be791d5-2577-4234-9289-6c870b08b7f8-kube-api-access-tx4qc\") pod \"node-ca-nlggj\" (UID: \"4be791d5-2577-4234-9289-6c870b08b7f8\") " pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.129549 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.129495 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6xp\" (UniqueName: \"kubernetes.io/projected/63158caf-bd4e-4285-9544-e5abb2424302-kube-api-access-9r6xp\") pod \"node-resolver-wz6vs\" (UID: \"63158caf-bd4e-4285-9544-e5abb2424302\") " pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.130953 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.130932 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mfk\" (UniqueName: \"kubernetes.io/projected/03fd2de7-bc46-4700-aa22-e765108fe445-kube-api-access-27mfk\") pod \"multus-5vxfr\" (UID: \"03fd2de7-bc46-4700-aa22-e765108fe445\") " pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.143992 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.143825 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wz6vs" Jul 23 23:12:59.152195 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.152157 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5vxfr" Jul 23 23:12:59.159400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.159169 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7n678" Jul 23 23:12:59.171656 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.171549 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wz6vs" event={"ID":"63158caf-bd4e-4285-9544-e5abb2424302","Type":"ContainerStarted","Data":"b77b6348e1a8a42a1420a24ca420aad6b2dcf7070d8272256ad93b9f952bb365"} Jul 23 23:12:59.176335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.176246 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"d3d7dce3e0224a71b7dfd90241d8e78fff9d0ad8afe1dab1006683149aa176c3"} Jul 23 23:12:59.178865 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:59.178838 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93b0ed8_c722_414d_8752_4dac337b6739.slice/crio-c466185066c2f9767ea4effa72d7aadac55bc3c871a01ffc067ed7d646225c58 WatchSource:0}: Error finding container c466185066c2f9767ea4effa72d7aadac55bc3c871a01ffc067ed7d646225c58: Status 404 returned error can't find the container with id c466185066c2f9767ea4effa72d7aadac55bc3c871a01ffc067ed7d646225c58 Jul 23 23:12:59.179634 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.179576 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6774" event={"ID":"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4","Type":"ContainerStarted","Data":"13a25b371be4942f3ffd14e3662c1ecddaae42db8bd55a41a4ed5fc6c1a8883c"} Jul 23 23:12:59.181481 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.181432 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" event={"ID":"ed436c2c06eb421aaa817d4e8a2cf780","Type":"ContainerStarted","Data":"2ac7edf0205d8bfc097ff8afd3b6af63a64fd9753abe212cb9b1a1a085668b15"} Jul 23 23:12:59.184231 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.184175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" event={"ID":"5914545796d27e5f7bd262f1db2d1af6","Type":"ContainerStarted","Data":"75053c5174db9f4972208ca6b138eb2d4472e55dc5cfdf36a774d919e1a293ae"} Jul 23 23:12:59.374845 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.374741 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" Jul 23 23:12:59.384940 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:59.384905 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e4d568_8bcb_43f9_87ea_6ee61e318bcf.slice/crio-0a66d2cf8ffc73aa36f302ef49fff56da3892984df46992af63666c051f2e252 WatchSource:0}: Error finding container 0a66d2cf8ffc73aa36f302ef49fff56da3892984df46992af63666c051f2e252: Status 404 returned error can't find the container with id 0a66d2cf8ffc73aa36f302ef49fff56da3892984df46992af63666c051f2e252 Jul 23 23:12:59.400097 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.400066 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stvz9" Jul 23 23:12:59.407328 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.407128 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j2s62" Jul 23 23:12:59.409516 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:59.409485 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c58c88_b124_4e13_8596_79c1031522fa.slice/crio-c1c300d856813ba2b37775473c917fccc026faf77daf3dafd3eb26e74f67e38f WatchSource:0}: Error finding container c1c300d856813ba2b37775473c917fccc026faf77daf3dafd3eb26e74f67e38f: Status 404 returned error can't find the container with id c1c300d856813ba2b37775473c917fccc026faf77daf3dafd3eb26e74f67e38f Jul 23 23:12:59.414941 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.414797 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nlggj" Jul 23 23:12:59.419926 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:59.419896 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6659ad44_c467_47ce_b25e_33dbcc41f0ba.slice/crio-f426bcc982fc31d64ba260e7c73b82cbdad0e846627c78c52114f5a1b4137f2f WatchSource:0}: Error finding container f426bcc982fc31d64ba260e7c73b82cbdad0e846627c78c52114f5a1b4137f2f: Status 404 returned error can't find the container with id f426bcc982fc31d64ba260e7c73b82cbdad0e846627c78c52114f5a1b4137f2f Jul 23 23:12:59.425059 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:12:59.425034 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be791d5_2577_4234_9289_6c870b08b7f8.slice/crio-978d3464fafaedc4f23ab77967d8778553434506a191fa7476bf514c3813b762 WatchSource:0}: Error finding container 978d3464fafaedc4f23ab77967d8778553434506a191fa7476bf514c3813b762: Status 404 returned error can't find the container with id 978d3464fafaedc4f23ab77967d8778553434506a191fa7476bf514c3813b762 Jul 23 23:12:59.517186 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.517153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:12:59.517349 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.517211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:12:59.517349 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.517304 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:12:59.517479 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.517365 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 23:12:59.517479 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.517381 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs podName:7933aa4d-a215-4a78-bcfd-ec12e3d6cf92 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:00.517359705 +0000 UTC m=+4.045636862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs") pod "network-metrics-daemon-xdbwg" (UID: "7933aa4d-a215-4a78-bcfd-ec12e3d6cf92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:12:59.517479 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.517385 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 23:12:59.517479 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.517401 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jfvrc for pod openshift-network-diagnostics/network-check-target-9bxbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:12:59.517479 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.517475 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc podName:620d6fc7-90b1-462c-8644-92d565a7d9e7 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:00.517459356 +0000 UTC m=+4.045736515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfvrc" (UniqueName: "kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc") pod "network-check-target-9bxbv" (UID: "620d6fc7-90b1-462c-8644-92d565a7d9e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:12:59.618651 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.618572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:12:59.618805 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.618695 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 23:12:59.618805 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:12:59.618756 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret podName:d90782c3-1c42-4715-804d-63e1d26969f2 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:00.618739142 +0000 UTC m=+4.147016298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret") pod "global-pull-secret-syncer-hc8bv" (UID: "d90782c3-1c42-4715-804d-63e1d26969f2") : object "kube-system"/"original-pull-secret" not registered Jul 23 23:12:59.976764 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.976576 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 23:07:58 +0000 UTC" deadline="2028-01-26 06:43:21.78648162 +0000 UTC" Jul 23 23:12:59.976764 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:12:59.976663 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13231h30m21.80982476s" Jul 23 23:13:00.160072 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.160035 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:00.160350 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.160317 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:00.194268 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.194231 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nlggj" event={"ID":"4be791d5-2577-4234-9289-6c870b08b7f8","Type":"ContainerStarted","Data":"978d3464fafaedc4f23ab77967d8778553434506a191fa7476bf514c3813b762"} Jul 23 23:13:00.195825 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.195794 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j2s62" event={"ID":"6659ad44-c467-47ce-b25e-33dbcc41f0ba","Type":"ContainerStarted","Data":"f426bcc982fc31d64ba260e7c73b82cbdad0e846627c78c52114f5a1b4137f2f"} Jul 23 23:13:00.199681 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.199656 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerStarted","Data":"c1c300d856813ba2b37775473c917fccc026faf77daf3dafd3eb26e74f67e38f"} Jul 23 23:13:00.205049 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.205018 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" event={"ID":"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf","Type":"ContainerStarted","Data":"0a66d2cf8ffc73aa36f302ef49fff56da3892984df46992af63666c051f2e252"} Jul 23 23:13:00.212537 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.212463 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7n678" event={"ID":"e93b0ed8-c722-414d-8752-4dac337b6739","Type":"ContainerStarted","Data":"c466185066c2f9767ea4effa72d7aadac55bc3c871a01ffc067ed7d646225c58"} Jul 23 23:13:00.216761 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.216439 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5vxfr" event={"ID":"03fd2de7-bc46-4700-aa22-e765108fe445","Type":"ContainerStarted","Data":"9178d2911599e0e1ed16ad1de471be7b08c503f20623ed094e0ed0c578b2be7e"} Jul 23 23:13:00.391056 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.390709 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.526370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.526469 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.526617 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.526682 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs podName:7933aa4d-a215-4a78-bcfd-ec12e3d6cf92 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:02.526662999 +0000 UTC m=+6.054940160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs") pod "network-metrics-daemon-xdbwg" (UID: "7933aa4d-a215-4a78-bcfd-ec12e3d6cf92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.527079 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.527096 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.527108 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jfvrc for pod openshift-network-diagnostics/network-check-target-9bxbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:00.527192 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.527153 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc podName:620d6fc7-90b1-462c-8644-92d565a7d9e7 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:02.527137283 +0000 UTC m=+6.055414431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfvrc" (UniqueName: "kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc") pod "network-check-target-9bxbv" (UID: "620d6fc7-90b1-462c-8644-92d565a7d9e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:00.627157 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.627124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:00.627337 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.627270 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:00.627403 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:00.627343 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret podName:d90782c3-1c42-4715-804d-63e1d26969f2 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:02.627322282 +0000 UTC m=+6.155599431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret") pod "global-pull-secret-syncer-hc8bv" (UID: "d90782c3-1c42-4715-804d-63e1d26969f2") : object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:00.977824 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.977780 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 23:07:58 +0000 UTC" deadline="2028-02-10 14:18:59.224100897 +0000 UTC" Jul 23 23:13:00.977824 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:00.977822 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13599h5m58.246283306s" Jul 23 23:13:01.162853 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:01.162821 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:01.163024 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:01.162961 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:01.163424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:01.163342 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:01.163526 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:01.163453 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:02.160164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:02.160133 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:02.160637 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.160345 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:02.543304 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:02.543200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:02.543304 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:02.543301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:02.543943 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.543398 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 23:13:02.543943 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.543438 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 23:13:02.543943 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.543455 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jfvrc for pod openshift-network-diagnostics/network-check-target-9bxbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:02.543943 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.543475 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:02.543943 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.543520 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc podName:620d6fc7-90b1-462c-8644-92d565a7d9e7 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:06.543500644 +0000 UTC m=+10.071777798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfvrc" (UniqueName: "kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc") pod "network-check-target-9bxbv" (UID: "620d6fc7-90b1-462c-8644-92d565a7d9e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:02.543943 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.543552 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs podName:7933aa4d-a215-4a78-bcfd-ec12e3d6cf92 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:06.543539829 +0000 UTC m=+10.071816981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs") pod "network-metrics-daemon-xdbwg" (UID: "7933aa4d-a215-4a78-bcfd-ec12e3d6cf92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:02.644340 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:02.643672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:02.644340 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.643881 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:02.644340 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:02.643943 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret podName:d90782c3-1c42-4715-804d-63e1d26969f2 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:06.643925784 +0000 UTC m=+10.172202932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret") pod "global-pull-secret-syncer-hc8bv" (UID: "d90782c3-1c42-4715-804d-63e1d26969f2") : object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:03.160384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:03.160353 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:03.160836 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:03.160511 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:03.160960 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:03.160939 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:03.161066 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:03.161036 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:04.160541 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:04.160458 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:04.161240 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:04.161206 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:05.161164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:05.160498 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:05.161164 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:05.160637 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:05.161164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:05.161036 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:05.161164 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:05.161127 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:06.160108 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:06.159846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:06.160108 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.159983 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:06.578213 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:06.577950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:06.578213 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:06.578033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:06.578213 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.578117 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:06.578213 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.578168 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 23:13:06.578213 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.578188 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs podName:7933aa4d-a215-4a78-bcfd-ec12e3d6cf92 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:14.578167713 +0000 UTC m=+18.106444862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs") pod "network-metrics-daemon-xdbwg" (UID: "7933aa4d-a215-4a78-bcfd-ec12e3d6cf92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:06.578213 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.578188 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 23:13:06.578213 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.578210 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jfvrc for pod openshift-network-diagnostics/network-check-target-9bxbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:06.578995 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.578242 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc podName:620d6fc7-90b1-462c-8644-92d565a7d9e7 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:14.578232309 +0000 UTC m=+18.106509455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfvrc" (UniqueName: "kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc") pod "network-check-target-9bxbv" (UID: "620d6fc7-90b1-462c-8644-92d565a7d9e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:06.678573 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:06.678541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:06.678721 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.678705 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:06.678771 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:06.678762 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret podName:d90782c3-1c42-4715-804d-63e1d26969f2 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:14.678747496 +0000 UTC m=+18.207024640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret") pod "global-pull-secret-syncer-hc8bv" (UID: "d90782c3-1c42-4715-804d-63e1d26969f2") : object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:07.160422 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:07.160329 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:07.160589 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:07.160464 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:07.160650 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:07.160609 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:07.160759 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:07.160715 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:08.159932 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:08.159898 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:08.160372 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:08.160007 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:09.159835 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:09.159803 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:09.160015 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:09.159803 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:09.160015 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:09.159957 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:09.160463 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:09.160071 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:10.160033 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:10.160011 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:10.160285 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:10.160103 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:11.160324 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:11.160292 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:11.160813 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:11.160338 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:11.160813 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:11.160445 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:11.160813 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:11.160493 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:12.159974 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:12.159937 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:12.160150 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:12.160076 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:13.159858 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:13.159820 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:13.160304 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:13.159820 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:13.160304 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:13.159964 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:13.160304 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:13.160043 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:14.159985 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:14.159953 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:14.160363 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.160055 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:14.640155 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:14.639916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:14.640329 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:14.640208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:14.640329 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.640055 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:14.640329 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.640294 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs podName:7933aa4d-a215-4a78-bcfd-ec12e3d6cf92 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:30.640273863 +0000 UTC m=+34.168551009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs") pod "network-metrics-daemon-xdbwg" (UID: "7933aa4d-a215-4a78-bcfd-ec12e3d6cf92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:14.640513 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.640335 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 23:13:14.640513 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.640355 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 23:13:14.640513 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.640367 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jfvrc for pod openshift-network-diagnostics/network-check-target-9bxbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:14.640513 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.640438 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc podName:620d6fc7-90b1-462c-8644-92d565a7d9e7 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:30.640419854 +0000 UTC m=+34.168697016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfvrc" (UniqueName: "kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc") pod "network-check-target-9bxbv" (UID: "620d6fc7-90b1-462c-8644-92d565a7d9e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:14.741623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:14.741586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:14.741792 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.741701 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:14.741792 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:14.741777 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret podName:d90782c3-1c42-4715-804d-63e1d26969f2 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:30.741761167 +0000 UTC m=+34.270038324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret") pod "global-pull-secret-syncer-hc8bv" (UID: "d90782c3-1c42-4715-804d-63e1d26969f2") : object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:15.160163 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:15.160127 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:15.160654 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:15.160278 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:15.160654 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:15.160325 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:15.160654 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:15.160428 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:16.160457 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:16.160424 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:16.160900 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:16.160549 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:17.161028 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:17.160997 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:17.161434 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:17.161122 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:17.161434 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:17.161164 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:17.161434 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:17.161220 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:18.160295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:18.160271 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:18.160387 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:18.160368 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:19.160178 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.159946 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:19.160906 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.159960 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:19.160906 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:19.160290 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:19.160906 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:19.160346 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:19.261071 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.261026 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" event={"ID":"ed436c2c06eb421aaa817d4e8a2cf780","Type":"ContainerStarted","Data":"1fb5524bc3b5c6e8aa95b721e6ee175d9b48bce282504968dcd22d67ea92db85"} Jul 23 23:13:19.262463 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.262436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nlggj" event={"ID":"4be791d5-2577-4234-9289-6c870b08b7f8","Type":"ContainerStarted","Data":"4faf5117efbedd4916b2c4f540f0220ac3506eeeb4ce1c78bfad977bb7259756"} Jul 23 23:13:19.263750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.263730 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j2s62" event={"ID":"6659ad44-c467-47ce-b25e-33dbcc41f0ba","Type":"ContainerStarted","Data":"fd9838295627c0fb30da2812f795874996179748b3c0b28a266c312e223d84b5"} Jul 23 23:13:19.265102 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.265077 2568 generic.go:356] "Generic (PLEG): container finished" podID="47c58c88-b124-4e13-8596-79c1031522fa" containerID="55451627bfed9a86f04f451030ddd352b97ce721194672de756dc85c8f2b9094" exitCode=0 Jul 23 23:13:19.265191 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.265103 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerDied","Data":"55451627bfed9a86f04f451030ddd352b97ce721194672de756dc85c8f2b9094"} Jul 23 23:13:19.266748 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.266731 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" event={"ID":"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf","Type":"ContainerStarted","Data":"46f68302f0cff6edafd14f60859a15af47b79c09ddf9078b9149640f5ac215ff"} Jul 23 23:13:19.268188 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.268167 2568 generic.go:356] "Generic (PLEG): container finished" podID="5914545796d27e5f7bd262f1db2d1af6" containerID="b74756b3853bdb9788295072af3a1a92ca47fb62410997158920cf5442c0e6d8" exitCode=0 Jul 23 23:13:19.268280 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.268241 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" event={"ID":"5914545796d27e5f7bd262f1db2d1af6","Type":"ContainerDied","Data":"b74756b3853bdb9788295072af3a1a92ca47fb62410997158920cf5442c0e6d8"} Jul 23 23:13:19.269754 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.269724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7n678" event={"ID":"e93b0ed8-c722-414d-8752-4dac337b6739","Type":"ContainerStarted","Data":"80c0908a21e8b9d8d661748907ef80ee742a22410003f940d6c92212027d3f5b"} Jul 23 23:13:19.271264 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.271245 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5vxfr" event={"ID":"03fd2de7-bc46-4700-aa22-e765108fe445","Type":"ContainerStarted","Data":"f4b1706b00e35a8093d04627e220643b2bc9a3e6ad5a7c36f2283d76e75bb53f"} Jul 23 23:13:19.272655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.272631 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wz6vs" event={"ID":"63158caf-bd4e-4285-9544-e5abb2424302","Type":"ContainerStarted","Data":"f24bbb87b6ce7c79163f61bdc8828327f2de983f25f4c4f04721d71f6f2374a9"} Jul 23 23:13:19.275556 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.275539 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:13:19.275863 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.275846 2568 generic.go:356] "Generic (PLEG): container finished" podID="afe83ceb-ece6-4972-b6ef-cf1671100c7b" containerID="a100161e4ae1f8a161dfc1dec1f51900f77eec333d955b8c61cc2ed56c9a24e9" exitCode=1 Jul 23 23:13:19.275945 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.275870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"6605c404f1466505e0b903622c09c7a81cad9395c68ab9c6d3f667bda06cbef9"} Jul 23 23:13:19.275945 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.275912 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"e03b6554a5f772af16a2506330346654a1faa3d3e8eeadb9aa30747f3f1954c2"} Jul 23 23:13:19.275945 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.275940 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"d5fa8f3dcdb09b046f02f20a10bd90a86c1e151399fb2bac7f1244a97ba2a223"} Jul 23 23:13:19.276111 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.275968 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"049341fdef2df759a626b492682da361c386248d9ce47f4e4e7562d21df8acc0"} Jul 23 23:13:19.276111 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.275999 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerDied","Data":"a100161e4ae1f8a161dfc1dec1f51900f77eec333d955b8c61cc2ed56c9a24e9"} Jul 23 23:13:19.276111 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.276028 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"8a2a9a9a6fe1cd4807108356e514bd46982ccabf747c3f2af7b54e238135c93e"} Jul 23 23:13:19.286021 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.285975 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-101.ec2.internal" podStartSLOduration=21.285958444 podStartE2EDuration="21.285958444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:19.273819806 +0000 UTC m=+22.802096975" watchObservedRunningTime="2026-07-23 23:13:19.285958444 +0000 UTC m=+22.814235612" Jul 23 23:13:19.299813 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.299779 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7n678" podStartSLOduration=3.298754827 podStartE2EDuration="22.299767271s" totalImagesPullingTime="19.001012444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:19.285949177 +0000 UTC m=+22.814226340" watchObservedRunningTime="2026-07-23 23:13:19.299767271 +0000 UTC m=+22.828044437" Jul 23 23:13:19.325798 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.325606 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j2s62" podStartSLOduration=3.399107291 podStartE2EDuration="22.325592137s" totalImagesPullingTime="18.926484846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:19.31412696 +0000 UTC m=+22.842404127" watchObservedRunningTime="2026-07-23 23:13:19.325592137 +0000 UTC m=+22.853869300" Jul 23 23:13:19.325979 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.325953 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nlggj" podStartSLOduration=3.405963478 podStartE2EDuration="22.325944485s" totalImagesPullingTime="18.919981007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:19.325876729 +0000 UTC m=+22.854153891" watchObservedRunningTime="2026-07-23 23:13:19.325944485 +0000 UTC m=+22.854221652" Jul 23 23:13:19.384944 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.384848 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wz6vs" podStartSLOduration=7.488902833 podStartE2EDuration="22.384827265s" totalImagesPullingTime="14.895924432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:19.340122242 +0000 UTC m=+22.868399409" watchObservedRunningTime="2026-07-23 23:13:19.384827265 +0000 UTC m=+22.913104432" Jul 23 23:13:19.403400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:19.403364 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5vxfr" podStartSLOduration=2.9043219479999998 podStartE2EDuration="22.403350222s" totalImagesPullingTime="19.499028274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:19.40276673 +0000 UTC m=+22.931043897" watchObservedRunningTime="2026-07-23 23:13:19.403350222 +0000 UTC m=+22.931627389" Jul 23 23:13:20.160259 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:20.160187 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:20.160717 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:20.160317 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:20.279792 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:20.279754 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" event={"ID":"5914545796d27e5f7bd262f1db2d1af6","Type":"ContainerStarted","Data":"f0b546925e5ff2c9227dba84e984fd12b0aa2c929dc998c0ea0b6db9a89f7f9e"} Jul 23 23:13:20.281068 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:20.281027 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6774" event={"ID":"e88142f1-c2db-42c8-bc93-4f5b09b5d8b4","Type":"ContainerStarted","Data":"58ebc8c712afb8134a4e4971e1089cab130b6ce27c174fe9cd54d6efbf504ccc"} Jul 23 23:13:20.307650 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:20.307607 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-101.ec2.internal" podStartSLOduration=22.30758824 podStartE2EDuration="22.30758824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:20.293005272 +0000 UTC m=+23.821282439" watchObservedRunningTime="2026-07-23 23:13:20.30758824 +0000 UTC m=+23.835865408" Jul 23 23:13:20.431958 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:20.431931 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 23:13:21.009044 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.008912 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T23:13:20.431953158Z","UUID":"d788aaeb-0146-4906-8a80-bec980f6255e","Handler":null,"Name":"","Endpoint":""} Jul 23 23:13:21.011381 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.011356 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 23:13:21.011535 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.011443 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 23:13:21.159576 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.159544 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:21.159764 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.159552 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:21.159764 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:21.159697 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:21.159764 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:21.159738 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:21.285974 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.285879 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" event={"ID":"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf","Type":"ContainerStarted","Data":"78b229d6c6d6fbf039a118a3b14bb04b87507c82b0462be335164d6578cd446d"} Jul 23 23:13:21.416150 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.416117 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7n678" Jul 23 23:13:21.416899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.416872 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7n678" Jul 23 23:13:21.432194 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:21.432151 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r6774" podStartSLOduration=5.159629293 podStartE2EDuration="24.432138773s" totalImagesPullingTime="19.27250948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:20.307336965 +0000 UTC m=+23.835614132" watchObservedRunningTime="2026-07-23 23:13:21.432138773 +0000 UTC m=+24.960415949" Jul 23 23:13:22.159631 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:22.159555 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:22.159774 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:22.159665 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:22.290342 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:22.290302 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" event={"ID":"c6e4d568-8bcb-43f9-87ea-6ee61e318bcf","Type":"ContainerStarted","Data":"9d6e56ea01c50c7d5680b5d950cce3370ea2c84399ad0d9d5d023368d4e964f0"} Jul 23 23:13:22.293776 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:22.293754 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:13:22.294279 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:22.294244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"798df9a4f3b2103b4215a94159eaec55eb5f1267ddc6f0743f8641ebc17f3452"} Jul 23 23:13:22.294401 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:22.294351 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7n678" Jul 23 23:13:22.294895 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:22.294878 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7n678" Jul 23 23:13:22.306161 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:22.306108 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzn5n" podStartSLOduration=3.933775695 podStartE2EDuration="25.306090189s" totalImagesPullingTime="21.372314494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:22.305867421 +0000 UTC m=+25.834144589" watchObservedRunningTime="2026-07-23 23:13:22.306090189 +0000 UTC m=+25.834367359" Jul 23 23:13:23.160303 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:23.160271 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:23.160486 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:23.160439 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:23.160486 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:23.160457 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:23.160616 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:23.160585 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:24.159596 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:24.159563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:24.159967 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:24.159689 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:25.160556 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.160317 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:25.161303 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.160369 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:25.161303 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:25.160598 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:25.161303 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:25.160676 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:25.301812 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.301780 2568 generic.go:356] "Generic (PLEG): container finished" podID="47c58c88-b124-4e13-8596-79c1031522fa" containerID="5c2b2d7c912ed7be16bb7199bb67476e7d1280de06212e0c60a0d8c7be53362b" exitCode=0 Jul 23 23:13:25.301974 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.301873 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerDied","Data":"5c2b2d7c912ed7be16bb7199bb67476e7d1280de06212e0c60a0d8c7be53362b"} Jul 23 23:13:25.305329 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.305290 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:13:25.305662 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.305643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"560eb2b731689bf0519ff924c0d67d11b51db24750e81c290f9e1e6f40e60352"} Jul 23 23:13:25.305891 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.305865 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:13:25.306003 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.305938 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:13:25.306003 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.305988 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:13:25.306101 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.306056 2568 scope.go:117] "RemoveContainer" containerID="a100161e4ae1f8a161dfc1dec1f51900f77eec333d955b8c61cc2ed56c9a24e9" Jul 23 23:13:25.321969 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.321948 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:13:25.322164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:25.322153 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:13:26.160626 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.160391 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:26.161247 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:26.160708 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:26.311699 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.311674 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:13:26.311981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.311960 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" event={"ID":"afe83ceb-ece6-4972-b6ef-cf1671100c7b","Type":"ContainerStarted","Data":"ef4e141b83506fe7f409057d249828a434b9a9d2a156a11e2e2ce7bf619059b5"} Jul 23 23:13:26.340569 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.340511 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" podStartSLOduration=10.023540044 podStartE2EDuration="29.340491816s" totalImagesPullingTime="19.316951772s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:26.340470562 +0000 UTC m=+29.868747729" watchObservedRunningTime="2026-07-23 23:13:26.340491816 +0000 UTC m=+29.868768985" Jul 23 23:13:26.526126 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.526100 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hc8bv"] Jul 23 23:13:26.526240 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.526217 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:26.526353 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:26.526330 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:26.530295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.530054 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bxbv"] Jul 23 23:13:26.530295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.530173 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:26.530295 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:26.530258 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:26.530864 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.530839 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xdbwg"] Jul 23 23:13:26.530975 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:26.530947 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:26.531062 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:26.531042 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:27.315380 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:27.315348 2568 generic.go:356] "Generic (PLEG): container finished" podID="47c58c88-b124-4e13-8596-79c1031522fa" containerID="fece3d9c6efb07e847efb880f5ccb92c928b477e6c55c0305f42a2ed84bba519" exitCode=0 Jul 23 23:13:27.315856 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:27.315441 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerDied","Data":"fece3d9c6efb07e847efb880f5ccb92c928b477e6c55c0305f42a2ed84bba519"} Jul 23 23:13:28.159769 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:28.159739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:28.159950 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:28.159837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:28.159950 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:28.159844 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:28.159950 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:28.159873 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:28.160109 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:28.159978 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:28.160109 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:28.160076 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:29.321971 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:29.321935 2568 generic.go:356] "Generic (PLEG): container finished" podID="47c58c88-b124-4e13-8596-79c1031522fa" containerID="b92fe6020ba6a35e0328e33d0aa8019dff16568b1907a81a6ab488e4a69f2e86" exitCode=0 Jul 23 23:13:29.322328 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:29.321991 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerDied","Data":"b92fe6020ba6a35e0328e33d0aa8019dff16568b1907a81a6ab488e4a69f2e86"} Jul 23 23:13:30.159514 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:30.159486 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:30.159514 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:30.159517 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:30.159795 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.159625 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xdbwg" podUID="7933aa4d-a215-4a78-bcfd-ec12e3d6cf92" Jul 23 23:13:30.159795 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:30.159646 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:30.159795 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.159737 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9bxbv" podUID="620d6fc7-90b1-462c-8644-92d565a7d9e7" Jul 23 23:13:30.159944 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.159814 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hc8bv" podUID="d90782c3-1c42-4715-804d-63e1d26969f2" Jul 23 23:13:30.664354 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:30.664322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:30.664837 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:30.664389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:30.664837 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.664496 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:30.664837 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.664533 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 23:13:30.664837 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.664552 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 23:13:30.664837 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.664566 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jfvrc for pod openshift-network-diagnostics/network-check-target-9bxbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:30.664837 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.664568 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs podName:7933aa4d-a215-4a78-bcfd-ec12e3d6cf92 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:02.664549165 +0000 UTC m=+66.192826315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs") pod "network-metrics-daemon-xdbwg" (UID: "7933aa4d-a215-4a78-bcfd-ec12e3d6cf92") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 23:13:30.664837 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.664617 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc podName:620d6fc7-90b1-462c-8644-92d565a7d9e7 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:02.664601083 +0000 UTC m=+66.192878245 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfvrc" (UniqueName: "kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc") pod "network-check-target-9bxbv" (UID: "620d6fc7-90b1-462c-8644-92d565a7d9e7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 23:13:30.764760 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:30.764711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:30.764932 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.764841 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:30.764932 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:30.764912 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret podName:d90782c3-1c42-4715-804d-63e1d26969f2 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:02.764894435 +0000 UTC m=+66.293171581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret") pod "global-pull-secret-syncer-hc8bv" (UID: "d90782c3-1c42-4715-804d-63e1d26969f2") : object "kube-system"/"original-pull-secret" not registered Jul 23 23:13:31.253755 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.253728 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-101.ec2.internal" event="NodeReady" Jul 23 23:13:31.253886 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.253847 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 23:13:31.285230 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.284569 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj"] Jul 23 23:13:31.288456 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.288432 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v"] Jul 23 23:13:31.288569 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.288499 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.290762 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.290316 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.290762 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.290351 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 23:13:31.290762 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.290567 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 23:13:31.290762 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.290614 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.290762 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.290752 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6hx5n\"" Jul 23 23:13:31.292556 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.292536 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8"] Jul 23 23:13:31.292675 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.292656 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.294655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.294637 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 23:13:31.294655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.294648 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hxqlc\"" Jul 23 23:13:31.294840 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.294689 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 23:13:31.294996 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.294977 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 23:13:31.297343 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.297319 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" Jul 23 23:13:31.298565 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.298544 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj"] Jul 23 23:13:31.299309 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.299264 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-cmxkl\"" Jul 23 23:13:31.299687 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.299561 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.299957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.299920 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.301041 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.300676 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v"] Jul 23 23:13:31.301264 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.301245 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 23:13:31.305996 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.305973 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8k2pn"] Jul 23 23:13:31.310067 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.310050 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn"] Jul 23 23:13:31.310221 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.310204 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.311794 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.311779 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 23:13:31.311892 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.311794 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 23:13:31.311892 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.311834 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m52bp\"" Jul 23 23:13:31.314654 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.314638 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl"] Jul 23 23:13:31.315060 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.315043 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:31.320786 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.320760 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf"] Jul 23 23:13:31.321353 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.321316 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.323691 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.323669 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.323947 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.323928 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 23:13:31.324164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.324146 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xwpqb\"" Jul 23 23:13:31.324515 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.324154 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.324972 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.324955 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 23:13:31.325329 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.325306 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 23:13:31.325763 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.325738 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jv797\"" Jul 23 23:13:31.327168 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.326884 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.327168 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.327057 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.334434 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.334386 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tg2ns"] Jul 23 23:13:31.334630 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.334612 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.337025 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.336949 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.337361 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.337165 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.337361 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.337317 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 23 23:13:31.338584 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.338545 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 23 23:13:31.339018 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.339000 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l24z5\"" Jul 23 23:13:31.341260 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.341240 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8"] Jul 23 23:13:31.341423 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.341280 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8k2pn"] Jul 23 23:13:31.341423 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.341304 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl"] Jul 23 23:13:31.341423 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.341324 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf"] Jul 23 23:13:31.341423 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.341340 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn"] Jul 23 23:13:31.341423 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.341366 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-f9q5n"] Jul 23 23:13:31.341839 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.341790 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.343635 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.343594 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-w627s\"" Jul 23 23:13:31.344377 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.343982 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.344377 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.344087 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 23:13:31.344377 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.344158 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.344377 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.344186 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 23:13:31.345377 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.345361 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-2cb84"] Jul 23 23:13:31.345800 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.345783 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.347511 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.347490 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h5xq2\"" Jul 23 23:13:31.347684 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.347668 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 23:13:31.347808 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.347673 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.347893 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.347849 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.348193 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.348172 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 23:13:31.349529 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.349484 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-78c6465647-5pnmv"] Jul 23 23:13:31.349904 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.349879 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:31.351089 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.351062 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 23:13:31.352469 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.352449 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 23:13:31.352701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.352684 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 23:13:31.352953 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.352904 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8zt7w\"" Jul 23 23:13:31.353296 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.353278 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-27mq6"] Jul 23 23:13:31.353401 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.353385 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.355175 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.355158 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 23 23:13:31.355441 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.355200 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 23 23:13:31.355587 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.355544 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 23 23:13:31.355668 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.355652 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.355818 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.355804 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.356050 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.355925 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hmlgd\"" Jul 23 23:13:31.356050 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.355925 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 23 23:13:31.356716 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.356696 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs"] Jul 23 23:13:31.357921 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.357899 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.360526 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.360502 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 23:13:31.361325 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.361304 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 23:13:31.361476 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.361459 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mmpgv\"" Jul 23 23:13:31.361760 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.361741 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.362064 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.362029 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.363712 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.363693 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tg2ns"] Jul 23 23:13:31.363815 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.363740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-f9q5n"] Jul 23 23:13:31.363815 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.363763 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-2cb84"] Jul 23 23:13:31.363815 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.363790 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs"] Jul 23 23:13:31.363815 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.363810 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-27mq6"] Jul 23 23:13:31.364014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.363841 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78c6465647-5pnmv"] Jul 23 23:13:31.364014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.363939 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" Jul 23 23:13:31.365851 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.365830 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-4tjcx\"" Jul 23 23:13:31.365932 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.365855 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 23:13:31.365976 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.365830 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 23:13:31.469655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469618 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:31.469655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7b7d3d4-4175-4601-862b-23b9048ab96b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.469861 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469686 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-serving-cert\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.469861 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.469861 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469799 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-tmp-dir\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.470014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.470014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469886 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:31.470014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.469935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-bound-sa-token\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.470168 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.470168 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888f0508-837a-4dd8-bd36-e5e0c752d917-tmp\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.470168 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxfm\" (UniqueName: \"kubernetes.io/projected/ff576690-faba-4356-9a7c-a66d0e79daf7-kube-api-access-8xxfm\") pod \"ingress-canary-27mq6\" (UID: \"ff576690-faba-4356-9a7c-a66d0e79daf7\") " pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.470317 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-trusted-ca\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.470317 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470193 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7b7d3d4-4175-4601-862b-23b9048ab96b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.470317 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470215 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzzwp\" (UniqueName: \"kubernetes.io/projected/b4ef8031-4d56-4616-932e-73bd80525891-kube-api-access-lzzwp\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.470317 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7bg\" (UniqueName: \"kubernetes.io/projected/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-kube-api-access-ht7bg\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.470317 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/888f0508-837a-4dd8-bd36-e5e0c752d917-snapshots\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.470317 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-certificates\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470315 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f74b68-8640-4e19-b356-b28c92c3f1e4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470365 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/888f0508-837a-4dd8-bd36-e5e0c752d917-serving-cert\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-stats-auth\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/888f0508-837a-4dd8-bd36-e5e0c752d917-service-ca-bundle\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470460 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hh6z\" (UniqueName: \"kubernetes.io/projected/30a1c6ad-fda1-4caa-b434-5ed525f489e2-kube-api-access-7hh6z\") pod \"volume-data-source-validator-549c6d54cf-mtjx8\" (UID: \"30a1c6ad-fda1-4caa-b434-5ed525f489e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470539 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbhj\" (UniqueName: \"kubernetes.io/projected/e6f74b68-8640-4e19-b356-b28c92c3f1e4-kube-api-access-7hbhj\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470566 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b4ef8031-4d56-4616-932e-73bd80525891-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470608 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-trusted-ca\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470640 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-config-volume\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470684 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcjx\" (UniqueName: \"kubernetes.io/projected/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-kube-api-access-mmcjx\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.470758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnzr\" (UniqueName: \"kubernetes.io/projected/3e6212a9-3d48-44e8-8a66-4f1b79d75910-kube-api-access-9qnzr\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470768 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-default-certificate\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6f74b68-8640-4e19-b356-b28c92c3f1e4-config\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff576690-faba-4356-9a7c-a66d0e79daf7-cert\") pod \"ingress-canary-27mq6\" (UID: \"ff576690-faba-4356-9a7c-a66d0e79daf7\") " pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfz86\" (UniqueName: \"kubernetes.io/projected/c7b7d3d4-4175-4601-862b-23b9048ab96b-kube-api-access-xfz86\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/272f9641-d504-4a5f-ae89-7d8b725eb91a-ca-trust-extracted\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470957 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7d7\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-kube-api-access-4f7d7\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.470982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/888f0508-837a-4dd8-bd36-e5e0c752d917-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471011 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwpd\" (UniqueName: \"kubernetes.io/projected/888f0508-837a-4dd8-bd36-e5e0c752d917-kube-api-access-qjwpd\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471037 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfff\" (UniqueName: \"kubernetes.io/projected/50fcdce6-7cd1-4b4b-93be-aa84c1aad178-kube-api-access-tpfff\") pod \"network-check-source-7d88cb55d8-x8khs\" (UID: \"50fcdce6-7cd1-4b4b-93be-aa84c1aad178\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471095 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-image-registry-private-configuration\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-config\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471150 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-installation-pull-secrets\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.471392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471207 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qr7z\" (UniqueName: \"kubernetes.io/projected/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-kube-api-access-6qr7z\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.472239 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.471240 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/893999eb-09a3-46c0-9f3e-8c7d3e008682-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:31.555888 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.555853 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:31.571728 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.571705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/888f0508-837a-4dd8-bd36-e5e0c752d917-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.571843 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.571736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwpd\" (UniqueName: \"kubernetes.io/projected/888f0508-837a-4dd8-bd36-e5e0c752d917-kube-api-access-qjwpd\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.571843 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.571755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfff\" (UniqueName: \"kubernetes.io/projected/50fcdce6-7cd1-4b4b-93be-aa84c1aad178-kube-api-access-tpfff\") pod \"network-check-source-7d88cb55d8-x8khs\" (UID: \"50fcdce6-7cd1-4b4b-93be-aa84c1aad178\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" Jul 23 23:13:31.571923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.571886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-image-registry-private-configuration\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.571923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.571916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-config\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.572018 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.571936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:31.572018 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.572011 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 23:13:31.572114 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572067 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-installation-pull-secrets\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.572114 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572101 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qr7z\" (UniqueName: \"kubernetes.io/projected/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-kube-api-access-6qr7z\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.572213 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572128 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/893999eb-09a3-46c0-9f3e-8c7d3e008682-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:31.572213 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.572167 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert podName:893999eb-09a3-46c0-9f3e-8c7d3e008682 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:32.072143322 +0000 UTC m=+35.600420476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-2cb84" (UID: "893999eb-09a3-46c0-9f3e-8c7d3e008682") : secret "networking-console-plugin-cert" not found Jul 23 23:13:31.572213 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572206 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7b7d3d4-4175-4601-862b-23b9048ab96b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.572370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-serving-cert\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.572370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.572370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-tmp-dir\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.572370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572335 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.572370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572359 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-bound-sa-token\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572436 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888f0508-837a-4dd8-bd36-e5e0c752d917-tmp\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572484 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxfm\" (UniqueName: \"kubernetes.io/projected/ff576690-faba-4356-9a7c-a66d0e79daf7-kube-api-access-8xxfm\") pod \"ingress-canary-27mq6\" (UID: \"ff576690-faba-4356-9a7c-a66d0e79daf7\") " pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-trusted-ca\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7b7d3d4-4175-4601-862b-23b9048ab96b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzzwp\" (UniqueName: \"kubernetes.io/projected/b4ef8031-4d56-4616-932e-73bd80525891-kube-api-access-lzzwp\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.572602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7bg\" (UniqueName: \"kubernetes.io/projected/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-kube-api-access-ht7bg\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/888f0508-837a-4dd8-bd36-e5e0c752d917-snapshots\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572652 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-certificates\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572680 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f74b68-8640-4e19-b356-b28c92c3f1e4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/888f0508-837a-4dd8-bd36-e5e0c752d917-serving-cert\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-stats-auth\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-tmp-dir\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/888f0508-837a-4dd8-bd36-e5e0c752d917-service-ca-bundle\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572855 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hh6z\" (UniqueName: \"kubernetes.io/projected/30a1c6ad-fda1-4caa-b434-5ed525f489e2-kube-api-access-7hh6z\") pod \"volume-data-source-validator-549c6d54cf-mtjx8\" (UID: \"30a1c6ad-fda1-4caa-b434-5ed525f489e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/888f0508-837a-4dd8-bd36-e5e0c752d917-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbhj\" (UniqueName: \"kubernetes.io/projected/e6f74b68-8640-4e19-b356-b28c92c3f1e4-kube-api-access-7hbhj\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b4ef8031-4d56-4616-932e-73bd80525891-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-trusted-ca\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572941 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-config\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.572973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.572945 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-config-volume\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcjx\" (UniqueName: \"kubernetes.io/projected/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-kube-api-access-mmcjx\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnzr\" (UniqueName: \"kubernetes.io/projected/3e6212a9-3d48-44e8-8a66-4f1b79d75910-kube-api-access-9qnzr\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-default-certificate\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573111 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/893999eb-09a3-46c0-9f3e-8c7d3e008682-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6f74b68-8640-4e19-b356-b28c92c3f1e4-config\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff576690-faba-4356-9a7c-a66d0e79daf7-cert\") pod \"ingress-canary-27mq6\" (UID: \"ff576690-faba-4356-9a7c-a66d0e79daf7\") " pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.573203 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.573262 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls podName:fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f nodeName:}" failed. No retries permitted until 2026-07-23 23:13:32.073245341 +0000 UTC m=+35.601522498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls") pod "dns-default-8k2pn" (UID: "fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f") : secret "dns-default-metrics-tls" not found Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573205 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfz86\" (UniqueName: \"kubernetes.io/projected/c7b7d3d4-4175-4601-862b-23b9048ab96b-kube-api-access-xfz86\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573297 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7b7d3d4-4175-4601-862b-23b9048ab96b-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/272f9641-d504-4a5f-ae89-7d8b725eb91a-ca-trust-extracted\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7d7\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-kube-api-access-4f7d7\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573452 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-config-volume\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.573534 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:31.573724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573589 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/272f9641-d504-4a5f-ae89-7d8b725eb91a-ca-trust-extracted\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.573600 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls podName:b4ef8031-4d56-4616-932e-73bd80525891 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:32.073584864 +0000 UTC m=+35.601862010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-prmzf" (UID: "b4ef8031-4d56-4616-932e-73bd80525891") : secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.573794 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.573948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/888f0508-837a-4dd8-bd36-e5e0c752d917-snapshots\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.573970 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls podName:3e6212a9-3d48-44e8-8a66-4f1b79d75910 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:32.073951782 +0000 UTC m=+35.602228929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls") pod "cluster-samples-operator-6748489656-dndjn" (UID: "3e6212a9-3d48-44e8-8a66-4f1b79d75910") : secret "samples-operator-tls" not found Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.573868 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.574011 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:32.074000668 +0000 UTC m=+35.602277813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : secret "router-metrics-certs-default" not found Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.574037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-certificates\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.574444 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.574092 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888f0508-837a-4dd8-bd36-e5e0c752d917-tmp\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.574836 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.574805 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-trusted-ca\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.574898 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.574809 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6f74b68-8640-4e19-b356-b28c92c3f1e4-config\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.574945 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.574924 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:32.074907981 +0000 UTC m=+35.603185149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : configmap references non-existent config key: service-ca.crt Jul 23 23:13:31.574995 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.574945 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 23:13:31.574995 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.574959 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v: secret "image-registry-tls" not found Jul 23 23:13:31.575076 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:31.575001 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls podName:272f9641-d504-4a5f-ae89-7d8b725eb91a nodeName:}" failed. No retries permitted until 2026-07-23 23:13:32.074986353 +0000 UTC m=+35.603263528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls") pod "image-registry-5fb7f4dfb5-2cj4v" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a") : secret "image-registry-tls" not found Jul 23 23:13:31.575491 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.575437 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/888f0508-837a-4dd8-bd36-e5e0c752d917-service-ca-bundle\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.576332 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.576289 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b4ef8031-4d56-4616-932e-73bd80525891-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.576471 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.576427 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-trusted-ca\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.577223 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.577198 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f74b68-8640-4e19-b356-b28c92c3f1e4-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.577223 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.577211 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7b7d3d4-4175-4601-862b-23b9048ab96b-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.577399 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.577383 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-default-certificate\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.577856 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.577824 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-serving-cert\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.578202 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.578173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-image-registry-private-configuration\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.578290 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.578242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff576690-faba-4356-9a7c-a66d0e79daf7-cert\") pod \"ingress-canary-27mq6\" (UID: \"ff576690-faba-4356-9a7c-a66d0e79daf7\") " pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.579014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.578715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-stats-auth\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.579014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.578944 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-installation-pull-secrets\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.579285 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.579239 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/888f0508-837a-4dd8-bd36-e5e0c752d917-serving-cert\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.583707 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.582919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnzr\" (UniqueName: \"kubernetes.io/projected/3e6212a9-3d48-44e8-8a66-4f1b79d75910-kube-api-access-9qnzr\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:31.583707 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.582930 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwpd\" (UniqueName: \"kubernetes.io/projected/888f0508-837a-4dd8-bd36-e5e0c752d917-kube-api-access-qjwpd\") pod \"insights-operator-5885fdb759-f9q5n\" (UID: \"888f0508-837a-4dd8-bd36-e5e0c752d917\") " pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.583707 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.583015 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qr7z\" (UniqueName: \"kubernetes.io/projected/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-kube-api-access-6qr7z\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:31.583707 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.583205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfff\" (UniqueName: \"kubernetes.io/projected/50fcdce6-7cd1-4b4b-93be-aa84c1aad178-kube-api-access-tpfff\") pod \"network-check-source-7d88cb55d8-x8khs\" (UID: \"50fcdce6-7cd1-4b4b-93be-aa84c1aad178\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" Jul 23 23:13:31.583707 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.583444 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxfm\" (UniqueName: \"kubernetes.io/projected/ff576690-faba-4356-9a7c-a66d0e79daf7-kube-api-access-8xxfm\") pod \"ingress-canary-27mq6\" (UID: \"ff576690-faba-4356-9a7c-a66d0e79daf7\") " pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.584528 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.584504 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-bound-sa-token\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.584819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.584775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7bg\" (UniqueName: \"kubernetes.io/projected/2038a8f5-2d28-4cb5-a57d-c026505bd5fc-kube-api-access-ht7bg\") pod \"console-operator-584dcb6b54-tg2ns\" (UID: \"2038a8f5-2d28-4cb5-a57d-c026505bd5fc\") " pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.586247 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.586223 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzzwp\" (UniqueName: \"kubernetes.io/projected/b4ef8031-4d56-4616-932e-73bd80525891-kube-api-access-lzzwp\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:31.586823 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.586798 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbhj\" (UniqueName: \"kubernetes.io/projected/e6f74b68-8640-4e19-b356-b28c92c3f1e4-kube-api-access-7hbhj\") pod \"service-ca-operator-5d6c75f84f-rhfnj\" (UID: \"e6f74b68-8640-4e19-b356-b28c92c3f1e4\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.586945 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.586927 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hh6z\" (UniqueName: \"kubernetes.io/projected/30a1c6ad-fda1-4caa-b434-5ed525f489e2-kube-api-access-7hh6z\") pod \"volume-data-source-validator-549c6d54cf-mtjx8\" (UID: \"30a1c6ad-fda1-4caa-b434-5ed525f489e2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" Jul 23 23:13:31.587010 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.586948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7d7\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-kube-api-access-4f7d7\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:31.587200 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.587174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfz86\" (UniqueName: \"kubernetes.io/projected/c7b7d3d4-4175-4601-862b-23b9048ab96b-kube-api-access-xfz86\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl\" (UID: \"c7b7d3d4-4175-4601-862b-23b9048ab96b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.588202 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.588118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcjx\" (UniqueName: \"kubernetes.io/projected/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-kube-api-access-mmcjx\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:31.603890 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.603865 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" Jul 23 23:13:31.621688 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.621660 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" Jul 23 23:13:31.655003 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.654871 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" Jul 23 23:13:31.679278 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.679035 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-f9q5n" Jul 23 23:13:31.680053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.679949 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:31.700566 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.700493 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-27mq6" Jul 23 23:13:31.712441 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.708519 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" Jul 23 23:13:31.807246 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.807214 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj"] Jul 23 23:13:31.820686 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:31.820628 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f74b68_8640_4e19_b356_b28c92c3f1e4.slice/crio-afc605c2a63764878cb305ab87657ab5df1063b5db33ca3d030f151d40929253 WatchSource:0}: Error finding container afc605c2a63764878cb305ab87657ab5df1063b5db33ca3d030f151d40929253: Status 404 returned error can't find the container with id afc605c2a63764878cb305ab87657ab5df1063b5db33ca3d030f151d40929253 Jul 23 23:13:31.827472 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.827246 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8"] Jul 23 23:13:31.849938 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:31.849895 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a1c6ad_fda1_4caa_b434_5ed525f489e2.slice/crio-1f5d7a1476770a4eff755ec6e4a4090cea3b64bea539b796b3f8140076d5afe5 WatchSource:0}: Error finding container 1f5d7a1476770a4eff755ec6e4a4090cea3b64bea539b796b3f8140076d5afe5: Status 404 returned error can't find the container with id 1f5d7a1476770a4eff755ec6e4a4090cea3b64bea539b796b3f8140076d5afe5 Jul 23 23:13:31.881044 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.880894 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl"] Jul 23 23:13:31.885360 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:31.885189 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b7d3d4_4175_4601_862b_23b9048ab96b.slice/crio-b0a5ce4eef4e297a45fb18b91d300aa03f41a4b927d533cf91a6f4e42013fc33 WatchSource:0}: Error finding container b0a5ce4eef4e297a45fb18b91d300aa03f41a4b927d533cf91a6f4e42013fc33: Status 404 returned error can't find the container with id b0a5ce4eef4e297a45fb18b91d300aa03f41a4b927d533cf91a6f4e42013fc33 Jul 23 23:13:31.921369 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.921335 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-f9q5n"] Jul 23 23:13:31.924714 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:31.924687 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod888f0508_837a_4dd8_bd36_e5e0c752d917.slice/crio-594d7006ebdb444cc62aeee7d2fea767a0c4c689595d597ebb212e9efa10108e WatchSource:0}: Error finding container 594d7006ebdb444cc62aeee7d2fea767a0c4c689595d597ebb212e9efa10108e: Status 404 returned error can't find the container with id 594d7006ebdb444cc62aeee7d2fea767a0c4c689595d597ebb212e9efa10108e Jul 23 23:13:31.932061 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.932034 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs"] Jul 23 23:13:31.934952 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:31.934927 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fcdce6_7cd1_4b4b_93be_aa84c1aad178.slice/crio-c74186e86891a532d70664e92bb49fac3c1109a9d8b4482c2a9ad5e1af552957 WatchSource:0}: Error finding container c74186e86891a532d70664e92bb49fac3c1109a9d8b4482c2a9ad5e1af552957: Status 404 returned error can't find the container with id c74186e86891a532d70664e92bb49fac3c1109a9d8b4482c2a9ad5e1af552957 Jul 23 23:13:31.939204 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.939183 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-27mq6"] Jul 23 23:13:31.942757 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:31.942736 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff576690_faba_4356_9a7c_a66d0e79daf7.slice/crio-14b2cb63c9c60d9f26719ed5cc15934f5cf9e8e451684fe1b5e613342c25535a WatchSource:0}: Error finding container 14b2cb63c9c60d9f26719ed5cc15934f5cf9e8e451684fe1b5e613342c25535a: Status 404 returned error can't find the container with id 14b2cb63c9c60d9f26719ed5cc15934f5cf9e8e451684fe1b5e613342c25535a Jul 23 23:13:31.953715 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:31.953687 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tg2ns"] Jul 23 23:13:31.956876 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:31.956848 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2038a8f5_2d28_4cb5_a57d_c026505bd5fc.slice/crio-baed1d7f6e449088abf8b9c29ccf7fa9a74991f1d620eb6d148fa0c412296c01 WatchSource:0}: Error finding container baed1d7f6e449088abf8b9c29ccf7fa9a74991f1d620eb6d148fa0c412296c01: Status 404 returned error can't find the container with id baed1d7f6e449088abf8b9c29ccf7fa9a74991f1d620eb6d148fa0c412296c01 Jul 23 23:13:32.085239 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.085208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:32.085424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.085288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:32.085424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.085325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:32.085424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.085368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:32.085424 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085374 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 23:13:32.085424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.085405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085458 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:33.085438209 +0000 UTC m=+36.613715367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : secret "router-metrics-certs-default" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085467 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085487 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v: secret "image-registry-tls" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.085496 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085502 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085525 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:33.085511405 +0000 UTC m=+36.613788561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : configmap references non-existent config key: service-ca.crt Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085530 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085547 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls podName:fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f nodeName:}" failed. No retries permitted until 2026-07-23 23:13:33.085537649 +0000 UTC m=+36.613814841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls") pod "dns-default-8k2pn" (UID: "fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f") : secret "dns-default-metrics-tls" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.085545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085578 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert podName:893999eb-09a3-46c0-9f3e-8c7d3e008682 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:33.085565132 +0000 UTC m=+36.613842277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-2cb84" (UID: "893999eb-09a3-46c0-9f3e-8c7d3e008682") : secret "networking-console-plugin-cert" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085583 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085609 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085614 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls podName:272f9641-d504-4a5f-ae89-7d8b725eb91a nodeName:}" failed. No retries permitted until 2026-07-23 23:13:33.085602917 +0000 UTC m=+36.613880066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls") pod "image-registry-5fb7f4dfb5-2cj4v" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a") : secret "image-registry-tls" not found Jul 23 23:13:32.085676 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085630 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls podName:b4ef8031-4d56-4616-932e-73bd80525891 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:33.08562166 +0000 UTC m=+36.613898812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-prmzf" (UID: "b4ef8031-4d56-4616-932e-73bd80525891") : secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:32.086211 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:32.085645 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls podName:3e6212a9-3d48-44e8-8a66-4f1b79d75910 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:33.085636558 +0000 UTC m=+36.613913706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls") pod "cluster-samples-operator-6748489656-dndjn" (UID: "3e6212a9-3d48-44e8-8a66-4f1b79d75910") : secret "samples-operator-tls" not found Jul 23 23:13:32.159880 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.159803 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:13:32.160031 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.159811 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:13:32.160031 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.159815 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:13:32.162220 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.162201 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 23:13:32.162364 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.162346 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 23:13:32.162474 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.162456 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rl54t\"" Jul 23 23:13:32.163216 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.163093 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-86k6d\"" Jul 23 23:13:32.266375 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.266124 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:32.335293 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.335258 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" event={"ID":"50fcdce6-7cd1-4b4b-93be-aa84c1aad178","Type":"ContainerStarted","Data":"c74186e86891a532d70664e92bb49fac3c1109a9d8b4482c2a9ad5e1af552957"} Jul 23 23:13:32.336890 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.336861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" event={"ID":"2038a8f5-2d28-4cb5-a57d-c026505bd5fc","Type":"ContainerStarted","Data":"baed1d7f6e449088abf8b9c29ccf7fa9a74991f1d620eb6d148fa0c412296c01"} Jul 23 23:13:32.338346 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.338321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" event={"ID":"c7b7d3d4-4175-4601-862b-23b9048ab96b","Type":"ContainerStarted","Data":"b0a5ce4eef4e297a45fb18b91d300aa03f41a4b927d533cf91a6f4e42013fc33"} Jul 23 23:13:32.339660 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.339634 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f9q5n" event={"ID":"888f0508-837a-4dd8-bd36-e5e0c752d917","Type":"ContainerStarted","Data":"594d7006ebdb444cc62aeee7d2fea767a0c4c689595d597ebb212e9efa10108e"} Jul 23 23:13:32.340919 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.340892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-27mq6" event={"ID":"ff576690-faba-4356-9a7c-a66d0e79daf7","Type":"ContainerStarted","Data":"14b2cb63c9c60d9f26719ed5cc15934f5cf9e8e451684fe1b5e613342c25535a"} Jul 23 23:13:32.342162 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.342133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" event={"ID":"30a1c6ad-fda1-4caa-b434-5ed525f489e2","Type":"ContainerStarted","Data":"1f5d7a1476770a4eff755ec6e4a4090cea3b64bea539b796b3f8140076d5afe5"} Jul 23 23:13:32.343536 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.343512 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" event={"ID":"e6f74b68-8640-4e19-b356-b28c92c3f1e4","Type":"ContainerStarted","Data":"afc605c2a63764878cb305ab87657ab5df1063b5db33ca3d030f151d40929253"} Jul 23 23:13:32.347269 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.347203 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:32.452878 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.452776 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:32.549250 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.549209 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:32.607276 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:32.607237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:33.097676 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.097630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.097705 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.097736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097771 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.097778 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097813 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097837 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls podName:fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f nodeName:}" failed. No retries permitted until 2026-07-23 23:13:35.097819779 +0000 UTC m=+38.626096943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls") pod "dns-default-8k2pn" (UID: "fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f") : secret "dns-default-metrics-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097870 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls podName:b4ef8031-4d56-4616-932e-73bd80525891 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:35.097852987 +0000 UTC m=+38.626130132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-prmzf" (UID: "b4ef8031-4d56-4616-932e-73bd80525891") : secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097881 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097897 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097927 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:35.097911853 +0000 UTC m=+38.626188998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : secret "router-metrics-certs-default" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097949 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls podName:3e6212a9-3d48-44e8-8a66-4f1b79d75910 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:35.097937787 +0000 UTC m=+38.626214932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls") pod "cluster-samples-operator-6748489656-dndjn" (UID: "3e6212a9-3d48-44e8-8a66-4f1b79d75910") : secret "samples-operator-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.097922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097983 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.097994 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v: secret "image-registry-tls" not found Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.098007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:33.098384 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.098031 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls podName:272f9641-d504-4a5f-ae89-7d8b725eb91a nodeName:}" failed. No retries permitted until 2026-07-23 23:13:35.098016324 +0000 UTC m=+38.626293490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls") pod "image-registry-5fb7f4dfb5-2cj4v" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a") : secret "image-registry-tls" not found Jul 23 23:13:33.099055 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.098080 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:35.098068839 +0000 UTC m=+38.626345984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : configmap references non-existent config key: service-ca.crt Jul 23 23:13:33.099055 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.098100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:33.099055 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.098197 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 23:13:33.099055 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:33.098239 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert podName:893999eb-09a3-46c0-9f3e-8c7d3e008682 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:35.098228566 +0000 UTC m=+38.626505725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-2cb84" (UID: "893999eb-09a3-46c0-9f3e-8c7d3e008682") : secret "networking-console-plugin-cert" not found Jul 23 23:13:33.258942 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.258910 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:33.361348 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.361257 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:33.628244 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.628144 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:33.727174 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:33.727142 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:35.115255 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.115216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.115289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.115332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.115364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115393 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115446 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.115455 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115484 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls podName:fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f nodeName:}" failed. No retries permitted until 2026-07-23 23:13:39.115461763 +0000 UTC m=+42.643738908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls") pod "dns-default-8k2pn" (UID: "fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f") : secret "dns-default-metrics-tls" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115516 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls podName:b4ef8031-4d56-4616-932e-73bd80525891 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:39.115498539 +0000 UTC m=+42.643775697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-prmzf" (UID: "b4ef8031-4d56-4616-932e-73bd80525891") : secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115572 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.115573 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115586 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v: secret "image-registry-tls" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115629 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls podName:272f9641-d504-4a5f-ae89-7d8b725eb91a nodeName:}" failed. No retries permitted until 2026-07-23 23:13:39.115615652 +0000 UTC m=+42.643892808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls") pod "image-registry-5fb7f4dfb5-2cj4v" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a") : secret "image-registry-tls" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.115649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115681 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:39.115670514 +0000 UTC m=+42.643947661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : configmap references non-existent config key: service-ca.crt Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115744 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 23:13:35.115853 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115747 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 23:13:35.116637 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115770 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert podName:893999eb-09a3-46c0-9f3e-8c7d3e008682 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:39.115762906 +0000 UTC m=+42.644040051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-2cb84" (UID: "893999eb-09a3-46c0-9f3e-8c7d3e008682") : secret "networking-console-plugin-cert" not found Jul 23 23:13:35.116637 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115782 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:39.115776182 +0000 UTC m=+42.644053327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : secret "router-metrics-certs-default" not found Jul 23 23:13:35.116637 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115825 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 23:13:35.116637 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:35.115868 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls podName:3e6212a9-3d48-44e8-8a66-4f1b79d75910 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:39.115850793 +0000 UTC m=+42.644127942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls") pod "cluster-samples-operator-6748489656-dndjn" (UID: "3e6212a9-3d48-44e8-8a66-4f1b79d75910") : secret "samples-operator-tls" not found Jul 23 23:13:35.923017 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:35.922839 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:36.024664 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:36.024565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:38.860465 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:38.860425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:38.925499 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:38.925460 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:39.157145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:39.157069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:39.157145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:39.157111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:39.157145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:39.157141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:39.157165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:39.157192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:39.157210 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157219 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157241 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v: secret "image-registry-tls" not found Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157251 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:47.157229032 +0000 UTC m=+50.685506178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : configmap references non-existent config key: service-ca.crt Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:39.157283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157290 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls podName:272f9641-d504-4a5f-ae89-7d8b725eb91a nodeName:}" failed. No retries permitted until 2026-07-23 23:13:47.15727418 +0000 UTC m=+50.685551343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls") pod "image-registry-5fb7f4dfb5-2cj4v" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a") : secret "image-registry-tls" not found Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157305 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157330 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157350 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert podName:893999eb-09a3-46c0-9f3e-8c7d3e008682 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:47.157338474 +0000 UTC m=+50.685615626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-2cb84" (UID: "893999eb-09a3-46c0-9f3e-8c7d3e008682") : secret "networking-console-plugin-cert" not found Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157363 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 23:13:39.157371 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157374 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls podName:b4ef8031-4d56-4616-932e-73bd80525891 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:47.157361801 +0000 UTC m=+50.685638960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-prmzf" (UID: "b4ef8031-4d56-4616-932e-73bd80525891") : secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:39.157812 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157404 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:47.157391116 +0000 UTC m=+50.685668268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : secret "router-metrics-certs-default" not found Jul 23 23:13:39.157812 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157433 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 23:13:39.157812 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157464 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 23:13:39.157812 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157467 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls podName:3e6212a9-3d48-44e8-8a66-4f1b79d75910 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:47.157458209 +0000 UTC m=+50.685735359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls") pod "cluster-samples-operator-6748489656-dndjn" (UID: "3e6212a9-3d48-44e8-8a66-4f1b79d75910") : secret "samples-operator-tls" not found Jul 23 23:13:39.157812 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:39.157495 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls podName:fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f nodeName:}" failed. No retries permitted until 2026-07-23 23:13:47.157488131 +0000 UTC m=+50.685765276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls") pod "dns-default-8k2pn" (UID: "fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f") : secret "dns-default-metrics-tls" not found Jul 23 23:13:40.126573 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:40.126519 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:40.226909 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:40.226861 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:41.116557 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.116520 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:41.221000 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.220926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:41.292844 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.292809 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:41.371077 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.371042 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" event={"ID":"50fcdce6-7cd1-4b4b-93be-aa84c1aad178","Type":"ContainerStarted","Data":"6bcdcf9fd5b9dc828c88de2a72e36992993ba79428b9be7694bdc86e9decf260"} Jul 23 23:13:41.372850 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.372815 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/0.log" Jul 23 23:13:41.372970 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.372868 2568 generic.go:356] "Generic (PLEG): container finished" podID="2038a8f5-2d28-4cb5-a57d-c026505bd5fc" containerID="28646f4212a4d7b52e134c6d4c59e72e75d2a7f6b0129a39eb3ac70748026397" exitCode=255 Jul 23 23:13:41.372970 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.372939 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" event={"ID":"2038a8f5-2d28-4cb5-a57d-c026505bd5fc","Type":"ContainerDied","Data":"28646f4212a4d7b52e134c6d4c59e72e75d2a7f6b0129a39eb3ac70748026397"} Jul 23 23:13:41.373192 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.373168 2568 scope.go:117] "RemoveContainer" containerID="28646f4212a4d7b52e134c6d4c59e72e75d2a7f6b0129a39eb3ac70748026397" Jul 23 23:13:41.374680 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.374655 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" event={"ID":"c7b7d3d4-4175-4601-862b-23b9048ab96b","Type":"ContainerStarted","Data":"52bc9f417e38fae9ba2cdd1298de4aba17eae45127ae23a50415f8ee0604c91b"} Jul 23 23:13:41.377817 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.377715 2568 generic.go:356] "Generic (PLEG): container finished" podID="47c58c88-b124-4e13-8596-79c1031522fa" containerID="430f561f6a90c3432542971b32fd7ebb335c52f074f6b2a68ec40318347bd3b8" exitCode=0 Jul 23 23:13:41.377817 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.377791 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerDied","Data":"430f561f6a90c3432542971b32fd7ebb335c52f074f6b2a68ec40318347bd3b8"} Jul 23 23:13:41.379244 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.379202 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f9q5n" event={"ID":"888f0508-837a-4dd8-bd36-e5e0c752d917","Type":"ContainerStarted","Data":"936cf4dc8d75091b83e437a6cb131d9e46e2f4b53afb30f2cf417fa70f468616"} Jul 23 23:13:41.380937 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.380897 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-27mq6" event={"ID":"ff576690-faba-4356-9a7c-a66d0e79daf7","Type":"ContainerStarted","Data":"188a804e0d7a65d514dc4df61242f7c1d44b33a374059691a3392f7685548ff9"} Jul 23 23:13:41.382427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.382386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" event={"ID":"30a1c6ad-fda1-4caa-b434-5ed525f489e2","Type":"ContainerStarted","Data":"e7c1da20eb2550cd9adacd3126d733ba3a18a1d26f746a04c8620ab322d86423"} Jul 23 23:13:41.384371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.384350 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" event={"ID":"e6f74b68-8640-4e19-b356-b28c92c3f1e4","Type":"ContainerStarted","Data":"1c894f312c73dc602439f81f7d76646e7acfc9b97d022e858f55b90e27b66533"} Jul 23 23:13:41.392490 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.391034 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x8khs" podStartSLOduration=4.485564002 podStartE2EDuration="13.391018195s" totalImagesPullingTime="8.905454193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:41.389557743 +0000 UTC m=+44.917834914" watchObservedRunningTime="2026-07-23 23:13:41.391018195 +0000 UTC m=+44.919295363" Jul 23 23:13:41.420350 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.420310 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" podStartSLOduration=4.870513368 podStartE2EDuration="13.420297068s" totalImagesPullingTime="8.5497837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:41.419621061 +0000 UTC m=+44.947898229" watchObservedRunningTime="2026-07-23 23:13:41.420297068 +0000 UTC m=+44.948574213" Jul 23 23:13:41.438721 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.438176 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-f9q5n" podStartSLOduration=4.927689206 podStartE2EDuration="13.438160809s" totalImagesPullingTime="8.510471603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:41.436652848 +0000 UTC m=+44.964930016" watchObservedRunningTime="2026-07-23 23:13:41.438160809 +0000 UTC m=+44.966437978" Jul 23 23:13:41.479514 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.478803 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" podStartSLOduration=4.867495111 podStartE2EDuration="13.478781424s" totalImagesPullingTime="8.611286313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:41.477070574 +0000 UTC m=+45.005347742" watchObservedRunningTime="2026-07-23 23:13:41.478781424 +0000 UTC m=+45.007058593" Jul 23 23:13:41.511896 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.511830 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-27mq6" podStartSLOduration=2.016083728 podStartE2EDuration="10.51172032s" totalImagesPullingTime="8.495636592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:41.511635799 +0000 UTC m=+45.039912967" watchObservedRunningTime="2026-07-23 23:13:41.51172032 +0000 UTC m=+45.039997488" Jul 23 23:13:41.528524 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.528477 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-mtjx8" podStartSLOduration=4.952012589 podStartE2EDuration="13.528458017s" totalImagesPullingTime="8.576445428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:41.527296726 +0000 UTC m=+45.055573896" watchObservedRunningTime="2026-07-23 23:13:41.528458017 +0000 UTC m=+45.056735164" Jul 23 23:13:41.634311 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.634281 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:41.680239 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.680215 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:41.680343 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.680307 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:41.739148 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:41.739076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:42.098831 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.098794 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p6zhr"] Jul 23 23:13:42.105744 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.105599 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:42.117076 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.117047 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p6zhr"] Jul 23 23:13:42.117243 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.117149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.119358 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.119338 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 23:13:42.119502 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.119341 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 23:13:42.119502 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.119345 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ckkjh\"" Jul 23 23:13:42.188750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.188726 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.188895 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.188792 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d0ade1-d285-489d-a7ad-0a1b3afd4943-data-volume\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.188895 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.188811 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tgg\" (UniqueName: \"kubernetes.io/projected/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-api-access-49tgg\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.188895 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.188838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79d0ade1-d285-489d-a7ad-0a1b3afd4943-crio-socket\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.189050 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.188926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.207345 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.207318 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:42.290280 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.290257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d0ade1-d285-489d-a7ad-0a1b3afd4943-data-volume\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.290628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.290284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tgg\" (UniqueName: \"kubernetes.io/projected/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-api-access-49tgg\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.290628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.290305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79d0ade1-d285-489d-a7ad-0a1b3afd4943-crio-socket\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.290628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.290420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.290628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.290478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.290628 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.290577 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 23:13:42.290628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.290592 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/79d0ade1-d285-489d-a7ad-0a1b3afd4943-crio-socket\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.290840 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.290630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/79d0ade1-d285-489d-a7ad-0a1b3afd4943-data-volume\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.290840 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.290643 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:42.790627299 +0000 UTC m=+46.318904443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : configmap "kube-rbac-proxy" not found Jul 23 23:13:42.290840 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.290580 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 23:13:42.290840 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.290680 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:42.79067136 +0000 UTC m=+46.318948505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : secret "insights-runtime-extractor-tls" not found Jul 23 23:13:42.299601 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.299576 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tgg\" (UniqueName: \"kubernetes.io/projected/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-api-access-49tgg\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.391701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.391614 2568 generic.go:356] "Generic (PLEG): container finished" podID="47c58c88-b124-4e13-8596-79c1031522fa" containerID="9e435d71a27dc74308178fefad74dd734acaa946bd405b2f62ed99e0ddffb8ea" exitCode=0 Jul 23 23:13:42.391701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.391688 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerDied","Data":"9e435d71a27dc74308178fefad74dd734acaa946bd405b2f62ed99e0ddffb8ea"} Jul 23 23:13:42.394816 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.394455 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:13:42.394964 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.394947 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/0.log" Jul 23 23:13:42.395076 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.394982 2568 generic.go:356] "Generic (PLEG): container finished" podID="2038a8f5-2d28-4cb5-a57d-c026505bd5fc" containerID="ece28ed5ef1476422f429a22cfdc856a06c6ee990509486f83727f0275e26ad8" exitCode=255 Jul 23 23:13:42.395076 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.395069 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" event={"ID":"2038a8f5-2d28-4cb5-a57d-c026505bd5fc","Type":"ContainerDied","Data":"ece28ed5ef1476422f429a22cfdc856a06c6ee990509486f83727f0275e26ad8"} Jul 23 23:13:42.395261 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.395128 2568 scope.go:117] "RemoveContainer" containerID="28646f4212a4d7b52e134c6d4c59e72e75d2a7f6b0129a39eb3ac70748026397" Jul 23 23:13:42.395710 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.395688 2568 scope.go:117] "RemoveContainer" containerID="ece28ed5ef1476422f429a22cfdc856a06c6ee990509486f83727f0275e26ad8" Jul 23 23:13:42.395928 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.395877 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tg2ns_openshift-console-operator(2038a8f5-2d28-4cb5-a57d-c026505bd5fc)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" podUID="2038a8f5-2d28-4cb5-a57d-c026505bd5fc" Jul 23 23:13:42.586059 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.586011 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:42.637347 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.637313 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:42.794823 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.794748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.794823 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.794797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:42.795002 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.794867 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 23:13:42.795002 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.794888 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 23:13:42.795002 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.794931 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:43.794917936 +0000 UTC m=+47.323195081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : secret "insights-runtime-extractor-tls" not found Jul 23 23:13:42.795002 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:42.794944 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:43.79493752 +0000 UTC m=+47.323214664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : configmap "kube-rbac-proxy" not found Jul 23 23:13:42.851447 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.851400 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:42.953222 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:42.953189 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:43.094950 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.094920 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp"] Jul 23 23:13:43.113779 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.113756 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp"] Jul 23 23:13:43.113922 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.113850 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" Jul 23 23:13:43.116107 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.116010 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 23:13:43.116107 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.116056 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 23:13:43.116107 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.116076 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ptbqp\"" Jul 23 23:13:43.135150 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.135124 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:43.193023 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.192992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:43.198110 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.198087 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kd2\" (UniqueName: \"kubernetes.io/projected/be93e488-fb0d-4505-8474-8a7763712fdd-kube-api-access-24kd2\") pod \"migrator-6865c9967-lrcjp\" (UID: \"be93e488-fb0d-4505-8474-8a7763712fdd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" Jul 23 23:13:43.299336 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.299306 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kd2\" (UniqueName: \"kubernetes.io/projected/be93e488-fb0d-4505-8474-8a7763712fdd-kube-api-access-24kd2\") pod \"migrator-6865c9967-lrcjp\" (UID: \"be93e488-fb0d-4505-8474-8a7763712fdd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" Jul 23 23:13:43.307099 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.307074 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kd2\" (UniqueName: \"kubernetes.io/projected/be93e488-fb0d-4505-8474-8a7763712fdd-kube-api-access-24kd2\") pod \"migrator-6865c9967-lrcjp\" (UID: \"be93e488-fb0d-4505-8474-8a7763712fdd\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" Jul 23 23:13:43.401821 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.401745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stvz9" event={"ID":"47c58c88-b124-4e13-8596-79c1031522fa","Type":"ContainerStarted","Data":"96d8faeb5085603657b12884e6df20eab52e39d7078772ce024f3bfc97c210c1"} Jul 23 23:13:43.403282 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.403264 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:13:43.403573 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.403558 2568 scope.go:117] "RemoveContainer" containerID="ece28ed5ef1476422f429a22cfdc856a06c6ee990509486f83727f0275e26ad8" Jul 23 23:13:43.403727 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:43.403708 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tg2ns_openshift-console-operator(2038a8f5-2d28-4cb5-a57d-c026505bd5fc)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" podUID="2038a8f5-2d28-4cb5-a57d-c026505bd5fc" Jul 23 23:13:43.422376 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.422339 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-stvz9" podStartSLOduration=8.976837878 podStartE2EDuration="46.422324833s" totalImagesPullingTime="37.445486955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:43.421666301 +0000 UTC m=+46.949943467" watchObservedRunningTime="2026-07-23 23:13:43.422324833 +0000 UTC m=+46.950601977" Jul 23 23:13:43.423708 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.423689 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" Jul 23 23:13:43.507955 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.507931 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wz6vs_63158caf-bd4e-4285-9544-e5abb2424302/dns-node-resolver/0.log" Jul 23 23:13:43.568295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.568239 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp"] Jul 23 23:13:43.568996 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.568952 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:43.571165 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:43.571142 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe93e488_fb0d_4505_8474_8a7763712fdd.slice/crio-5a480361c8a169ca96902aa4e98f5d625d63825f8b9cb7ed6331eadff442c551 WatchSource:0}: Error finding container 5a480361c8a169ca96902aa4e98f5d625d63825f8b9cb7ed6331eadff442c551: Status 404 returned error can't find the container with id 5a480361c8a169ca96902aa4e98f5d625d63825f8b9cb7ed6331eadff442c551 Jul 23 23:13:43.667711 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.667639 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:43.803713 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.803688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:43.803851 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:43.803741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:43.803896 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:43.803861 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 23:13:43.803937 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:43.803911 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:45.803898517 +0000 UTC m=+49.332175662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : secret "insights-runtime-extractor-tls" not found Jul 23 23:13:43.803937 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:43.803919 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 23:13:43.804013 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:43.803970 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:45.803957499 +0000 UTC m=+49.332234645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : configmap "kube-rbac-proxy" not found Jul 23 23:13:44.304974 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.304946 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nlggj_4be791d5-2577-4234-9289-6c870b08b7f8/node-ca/0.log" Jul 23 23:13:44.409666 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.409627 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" event={"ID":"be93e488-fb0d-4505-8474-8a7763712fdd","Type":"ContainerStarted","Data":"5a480361c8a169ca96902aa4e98f5d625d63825f8b9cb7ed6331eadff442c551"} Jul 23 23:13:44.449940 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.449910 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-ndzz5"] Jul 23 23:13:44.462261 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.462239 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-ndzz5"] Jul 23 23:13:44.462401 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.462335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.464535 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.464510 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 23:13:44.466110 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.465555 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 23:13:44.466493 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.466474 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 23:13:44.466765 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.466740 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-79v2x\"" Jul 23 23:13:44.468082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.466842 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 23:13:44.486122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.486094 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:44.510182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.510153 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/15802120-561a-4c09-bd63-6b92d01024fd-signing-cabundle\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.510342 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.510311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/15802120-561a-4c09-bd63-6b92d01024fd-signing-key\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.510449 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.510359 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596qm\" (UniqueName: \"kubernetes.io/projected/15802120-561a-4c09-bd63-6b92d01024fd-kube-api-access-596qm\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.539637 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.539613 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:44.611750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.611721 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596qm\" (UniqueName: \"kubernetes.io/projected/15802120-561a-4c09-bd63-6b92d01024fd-kube-api-access-596qm\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.611899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.611889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/15802120-561a-4c09-bd63-6b92d01024fd-signing-cabundle\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.611963 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.611923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/15802120-561a-4c09-bd63-6b92d01024fd-signing-key\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.612581 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.612563 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/15802120-561a-4c09-bd63-6b92d01024fd-signing-cabundle\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.614891 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.614869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/15802120-561a-4c09-bd63-6b92d01024fd-signing-key\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.620989 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.620969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-596qm\" (UniqueName: \"kubernetes.io/projected/15802120-561a-4c09-bd63-6b92d01024fd-kube-api-access-596qm\") pod \"service-ca-658d7b7c78-ndzz5\" (UID: \"15802120-561a-4c09-bd63-6b92d01024fd\") " pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.775644 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.775611 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" Jul 23 23:13:44.906024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:44.905891 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-27mq6_ff576690-faba-4356-9a7c-a66d0e79daf7/serve-healthcheck-canary/0.log" Jul 23 23:13:45.131223 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:45.131193 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-ndzz5"] Jul 23 23:13:45.134619 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:45.134592 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15802120_561a_4c09_bd63_6b92d01024fd.slice/crio-37d3616b304e58c3da1ab7c373f94aa78efd13c0b1eef6af155d8fb865e14fa5 WatchSource:0}: Error finding container 37d3616b304e58c3da1ab7c373f94aa78efd13c0b1eef6af155d8fb865e14fa5: Status 404 returned error can't find the container with id 37d3616b304e58c3da1ab7c373f94aa78efd13c0b1eef6af155d8fb865e14fa5 Jul 23 23:13:45.414550 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:45.414470 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" event={"ID":"15802120-561a-4c09-bd63-6b92d01024fd","Type":"ContainerStarted","Data":"2afd3e2b905cd96126908e1597367ce8625333cec8d632d882608d7c694f170b"} Jul 23 23:13:45.414550 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:45.414525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" event={"ID":"15802120-561a-4c09-bd63-6b92d01024fd","Type":"ContainerStarted","Data":"37d3616b304e58c3da1ab7c373f94aa78efd13c0b1eef6af155d8fb865e14fa5"} Jul 23 23:13:45.425807 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:45.425783 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:45.431976 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:45.431934 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-ndzz5" podStartSLOduration=1.431920708 podStartE2EDuration="1.431920708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:45.430907127 +0000 UTC m=+48.959184294" watchObservedRunningTime="2026-07-23 23:13:45.431920708 +0000 UTC m=+48.960197874" Jul 23 23:13:45.822927 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:45.822897 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:45.823053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:45.823021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:45.823094 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:45.823051 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 23:13:45.823125 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:45.823097 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 23:13:45.823125 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:45.823118 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:49.823098962 +0000 UTC m=+53.351376112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : secret "insights-runtime-extractor-tls" not found Jul 23 23:13:45.823196 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:45.823137 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:49.823127775 +0000 UTC m=+53.351404926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : configmap "kube-rbac-proxy" not found Jul 23 23:13:46.419877 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:46.419846 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" event={"ID":"be93e488-fb0d-4505-8474-8a7763712fdd","Type":"ContainerStarted","Data":"2670e68c526cf22d3df8fdef58ef22baa8c1d198e48ad0a526739fa1e685ffcd"} Jul 23 23:13:46.420251 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:46.419903 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" event={"ID":"be93e488-fb0d-4505-8474-8a7763712fdd","Type":"ContainerStarted","Data":"5568c32a84ca3c428fa0e1274bdc47bb0aa4683b9bb02361c88d95700dccb6db"} Jul 23 23:13:46.437620 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:46.437575 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-lrcjp" podStartSLOduration=1.546939114 podStartE2EDuration="3.437559379s" totalImagesPullingTime="1.890620265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:46.436443311 +0000 UTC m=+49.964720476" watchObservedRunningTime="2026-07-23 23:13:46.437559379 +0000 UTC m=+49.965836576" Jul 23 23:13:46.710431 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:46.710331 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:46.899821 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:46.899786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:47.057727 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.057650 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:47.237799 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.237758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:47.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.237829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:13:47.237997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.237906 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:13:47.237997 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.237950 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:03.237925625 +0000 UTC m=+66.766202771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : configmap references non-existent config key: service-ca.crt Jul 23 23:13:47.238190 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238008 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 23:13:47.238190 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.238017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:13:47.238190 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.238049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:13:47.238190 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238083 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 23:13:47.238190 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.238091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:13:47.238190 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238119 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert podName:893999eb-09a3-46c0-9f3e-8c7d3e008682 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:03.23810386 +0000 UTC m=+66.766381012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-2cb84" (UID: "893999eb-09a3-46c0-9f3e-8c7d3e008682") : secret "networking-console-plugin-cert" not found Jul 23 23:13:47.238190 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238169 2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238207 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs podName:7a47b482-7b1c-4c44-aa8c-23e5b0d262c6 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:03.23819432 +0000 UTC m=+66.766471471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs") pod "router-default-78c6465647-5pnmv" (UID: "7a47b482-7b1c-4c44-aa8c-23e5b0d262c6") : secret "router-metrics-certs-default" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.238202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238261 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238274 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v: secret "image-registry-tls" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238303 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls podName:272f9641-d504-4a5f-ae89-7d8b725eb91a nodeName:}" failed. No retries permitted until 2026-07-23 23:14:03.238293706 +0000 UTC m=+66.766570857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls") pod "image-registry-5fb7f4dfb5-2cj4v" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a") : secret "image-registry-tls" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238356 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238383 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls podName:3e6212a9-3d48-44e8-8a66-4f1b79d75910 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:03.238373597 +0000 UTC m=+66.766650746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls") pod "cluster-samples-operator-6748489656-dndjn" (UID: "3e6212a9-3d48-44e8-8a66-4f1b79d75910") : secret "samples-operator-tls" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238444 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238473 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls podName:b4ef8031-4d56-4616-932e-73bd80525891 nodeName:}" failed. No retries permitted until 2026-07-23 23:14:03.238463904 +0000 UTC m=+66.766741057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-prmzf" (UID: "b4ef8031-4d56-4616-932e-73bd80525891") : secret "cluster-monitoring-operator-tls" not found Jul 23 23:13:47.238534 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:47.238487 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls podName:fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f nodeName:}" failed. No retries permitted until 2026-07-23 23:14:03.238479807 +0000 UTC m=+66.766756958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls") pod "dns-default-8k2pn" (UID: "fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f") : secret "dns-default-metrics-tls" not found Jul 23 23:13:47.351453 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.351403 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:47.523227 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.523190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:47.623936 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.623847 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:47.811353 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.811313 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:47.936980 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:47.936882 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:48.486247 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:48.486216 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:48.618800 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:48.618770 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:49.220394 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:49.220360 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:49.451774 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:49.451745 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:49.623280 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:49.623242 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:49.865043 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:49.865002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:49.865214 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:49.865062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:49.865214 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:49.865157 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 23:13:49.865214 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:13:49.865211 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls podName:79d0ade1-d285-489d-a7ad-0a1b3afd4943 nodeName:}" failed. No retries permitted until 2026-07-23 23:13:57.865195788 +0000 UTC m=+61.393472934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6zhr" (UID: "79d0ade1-d285-489d-a7ad-0a1b3afd4943") : secret "insights-runtime-extractor-tls" not found Jul 23 23:13:49.866181 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:49.866160 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/79d0ade1-d285-489d-a7ad-0a1b3afd4943-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:50.061507 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:50.061392 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:50.408968 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:50.408932 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:50.512267 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:50.512236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:50.613084 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:50.613047 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:51.213124 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:51.213089 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:51.414143 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:51.414108 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:51.514946 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:51.514860 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:51.614937 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:51.614898 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:51.680437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:51.680404 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:51.680568 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:51.680464 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:51.680783 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:51.680772 2568 scope.go:117] "RemoveContainer" containerID="ece28ed5ef1476422f429a22cfdc856a06c6ee990509486f83727f0275e26ad8" Jul 23 23:13:52.007620 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.007586 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:52.108906 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.108875 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:52.209875 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.209838 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:52.419784 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.419738 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:52.446210 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.446186 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:13:52.446354 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.446257 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" event={"ID":"2038a8f5-2d28-4cb5-a57d-c026505bd5fc","Type":"ContainerStarted","Data":"2793e4b540d82c8b3ed4035aa45b7e7fdb257903a8ef31358ac78505bd64c2da"} Jul 23 23:13:52.446596 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.446573 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:52.451171 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.451152 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" Jul 23 23:13:52.461490 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.461450 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-tg2ns" podStartSLOduration=15.97537547 podStartE2EDuration="24.461399604s" totalImagesPullingTime="8.486024134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:13:52.461156717 +0000 UTC m=+55.989433882" watchObservedRunningTime="2026-07-23 23:13:52.461399604 +0000 UTC m=+55.989676750" Jul 23 23:13:52.598838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.598804 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:52.701825 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.701753 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:52.753687 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.753657 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:52.903170 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:52.903121 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.130107 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.130083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.230761 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.230729 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.330903 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.330863 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.431922 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.431845 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.532177 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.532147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.632956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.632917 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.733056 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.732981 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.834735 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.834705 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:53.936386 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:53.936353 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:54.109592 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:54.109565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:54.318207 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:54.318177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:54.807499 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:54.807465 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:55.005060 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:55.005031 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:55.141945 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:55.141908 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:55.241352 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:55.241318 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:55.405780 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:55.405698 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:55.611984 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:55.611953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:56.202216 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:56.202179 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:56.607241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:56.607193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:56.804233 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:56.804200 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:57.035142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:57.035076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:57.206680 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:57.206641 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:57.325775 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:57.325754 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zb8ln" Jul 23 23:13:57.608944 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:57.608844 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:57.831871 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:57.831838 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:13:57.942401 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:57.942326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:57.944549 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:57.944533 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/79d0ade1-d285-489d-a7ad-0a1b3afd4943-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6zhr\" (UID: \"79d0ade1-d285-489d-a7ad-0a1b3afd4943\") " pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:58.032044 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:58.032015 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ckkjh\"" Jul 23 23:13:58.041476 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:58.041453 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p6zhr" Jul 23 23:13:58.163226 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:58.163199 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p6zhr"] Jul 23 23:13:58.165964 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:13:58.165940 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d0ade1_d285_489d_a7ad_0a1b3afd4943.slice/crio-f1e44f6a87b92f3f7295c1bf1692f95ff61da9cbb80118777131d0ccfbb7b5f4 WatchSource:0}: Error finding container f1e44f6a87b92f3f7295c1bf1692f95ff61da9cbb80118777131d0ccfbb7b5f4: Status 404 returned error can't find the container with id f1e44f6a87b92f3f7295c1bf1692f95ff61da9cbb80118777131d0ccfbb7b5f4 Jul 23 23:13:58.464278 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:13:58.464245 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6zhr" event={"ID":"79d0ade1-d285-489d-a7ad-0a1b3afd4943","Type":"ContainerStarted","Data":"f1e44f6a87b92f3f7295c1bf1692f95ff61da9cbb80118777131d0ccfbb7b5f4"} Jul 23 23:14:01.532503 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:01.532460 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:02.690661 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.690623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:14:02.691124 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.690699 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:14:02.692604 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.692578 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 23:14:02.693466 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.693440 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfvrc\" (UniqueName: \"kubernetes.io/projected/620d6fc7-90b1-462c-8644-92d565a7d9e7-kube-api-access-jfvrc\") pod \"network-check-target-9bxbv\" (UID: \"620d6fc7-90b1-462c-8644-92d565a7d9e7\") " pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:14:02.703316 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.703299 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7933aa4d-a215-4a78-bcfd-ec12e3d6cf92-metrics-certs\") pod \"network-metrics-daemon-xdbwg\" (UID: \"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92\") " pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:14:02.775361 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.775325 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-86k6d\"" Jul 23 23:14:02.783984 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.783962 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xdbwg" Jul 23 23:14:02.786843 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.786823 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rl54t\"" Jul 23 23:14:02.791829 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.791809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:14:02.793637 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.793620 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 23:14:02.795901 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.795885 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:14:02.805134 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:02.805083 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90782c3-1c42-4715-804d-63e1d26969f2-original-pull-secret\") pod \"global-pull-secret-syncer-hc8bv\" (UID: \"d90782c3-1c42-4715-804d-63e1d26969f2\") " pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:14:03.079893 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.079858 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hc8bv" Jul 23 23:14:03.092485 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.092461 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9bxbv"] Jul 23 23:14:03.095790 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:03.095766 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620d6fc7_90b1_462c_8644_92d565a7d9e7.slice/crio-36df0683b49a6f9ea64c36adacb20a524cc8f498478c6afa182844ab776b5ae5 WatchSource:0}: Error finding container 36df0683b49a6f9ea64c36adacb20a524cc8f498478c6afa182844ab776b5ae5: Status 404 returned error can't find the container with id 36df0683b49a6f9ea64c36adacb20a524cc8f498478c6afa182844ab776b5ae5 Jul 23 23:14:03.108837 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.108814 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xdbwg"] Jul 23 23:14:03.111941 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:03.111915 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7933aa4d_a215_4a78_bcfd_ec12e3d6cf92.slice/crio-5648eeb496c55d40b60d8eb20a0e2bb527ca2479c7f600efd3ea234262ddf478 WatchSource:0}: Error finding container 5648eeb496c55d40b60d8eb20a0e2bb527ca2479c7f600efd3ea234262ddf478: Status 404 returned error can't find the container with id 5648eeb496c55d40b60d8eb20a0e2bb527ca2479c7f600efd3ea234262ddf478 Jul 23 23:14:03.132982 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.132764 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:03.212023 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.211960 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hc8bv"] Jul 23 23:14:03.214814 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:03.214787 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90782c3_1c42_4715_804d_63e1d26969f2.slice/crio-7d9a049bac1c496953d1b9554d1c27d3060648e0df90d0833a896dd2c6be6f36 WatchSource:0}: Error finding container 7d9a049bac1c496953d1b9554d1c27d3060648e0df90d0833a896dd2c6be6f36: Status 404 returned error can't find the container with id 7d9a049bac1c496953d1b9554d1c27d3060648e0df90d0833a896dd2c6be6f36 Jul 23 23:14:03.215496 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.215463 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:03.299582 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.299551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:03.299740 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.299596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:14:03.299796 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.299775 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:14:03.299835 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.299825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:14:03.299879 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.299852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:14:03.299932 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.299886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:03.299978 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.299966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:14:03.300319 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.300285 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-service-ca-bundle\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:03.302159 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.302137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f-metrics-tls\") pod \"dns-default-8k2pn\" (UID: \"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f\") " pod="openshift-dns/dns-default-8k2pn" Jul 23 23:14:03.302353 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.302229 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b4ef8031-4d56-4616-932e-73bd80525891-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-prmzf\" (UID: \"b4ef8031-4d56-4616-932e-73bd80525891\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:14:03.302451 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.302370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/893999eb-09a3-46c0-9f3e-8c7d3e008682-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-2cb84\" (UID: \"893999eb-09a3-46c0-9f3e-8c7d3e008682\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:14:03.302517 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.302454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3e6212a9-3d48-44e8-8a66-4f1b79d75910-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-dndjn\" (UID: \"3e6212a9-3d48-44e8-8a66-4f1b79d75910\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:14:03.302704 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.302684 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47b482-7b1c-4c44-aa8c-23e5b0d262c6-metrics-certs\") pod \"router-default-78c6465647-5pnmv\" (UID: \"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6\") " pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:03.302783 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.302763 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"image-registry-5fb7f4dfb5-2cj4v\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:14:03.412588 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.412559 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hxqlc\"" Jul 23 23:14:03.420996 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.420980 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:14:03.435823 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.435798 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m52bp\"" Jul 23 23:14:03.444294 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.444275 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8k2pn" Jul 23 23:14:03.448078 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.448056 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xwpqb\"" Jul 23 23:14:03.457475 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.457450 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" Jul 23 23:14:03.466331 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.466273 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-l24z5\"" Jul 23 23:14:03.474771 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.474728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" Jul 23 23:14:03.487288 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.487182 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hc8bv" event={"ID":"d90782c3-1c42-4715-804d-63e1d26969f2","Type":"ContainerStarted","Data":"7d9a049bac1c496953d1b9554d1c27d3060648e0df90d0833a896dd2c6be6f36"} Jul 23 23:14:03.488045 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.487836 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8zt7w\"" Jul 23 23:14:03.490531 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.490251 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bxbv" event={"ID":"620d6fc7-90b1-462c-8644-92d565a7d9e7","Type":"ContainerStarted","Data":"cf8077a1e7b344be704badbaa88d13ff35312bdb393c729040349d83afa603df"} Jul 23 23:14:03.490531 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.490299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9bxbv" event={"ID":"620d6fc7-90b1-462c-8644-92d565a7d9e7","Type":"ContainerStarted","Data":"36df0683b49a6f9ea64c36adacb20a524cc8f498478c6afa182844ab776b5ae5"} Jul 23 23:14:03.490531 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.490471 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:14:03.492022 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.491954 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xdbwg" event={"ID":"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92","Type":"ContainerStarted","Data":"5648eeb496c55d40b60d8eb20a0e2bb527ca2479c7f600efd3ea234262ddf478"} Jul 23 23:14:03.494531 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.494503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6zhr" event={"ID":"79d0ade1-d285-489d-a7ad-0a1b3afd4943","Type":"ContainerStarted","Data":"4433ad82b8260c94193bdb2ad0634569fe61234605e4b12024871a8f89cb465c"} Jul 23 23:14:03.494682 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.494634 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hmlgd\"" Jul 23 23:14:03.496937 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.496852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" Jul 23 23:14:03.507611 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.507399 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:03.508919 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.508767 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9bxbv" podStartSLOduration=66.508743546 podStartE2EDuration="1m6.508743546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:03.507622923 +0000 UTC m=+67.035900102" watchObservedRunningTime="2026-07-23 23:14:03.508743546 +0000 UTC m=+67.037020714" Jul 23 23:14:03.592784 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.592639 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v"] Jul 23 23:14:03.636651 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.636620 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8k2pn"] Jul 23 23:14:03.699240 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.699159 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn"] Jul 23 23:14:03.724322 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.723395 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf"] Jul 23 23:14:03.730033 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:03.729996 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ef8031_4d56_4616_932e_73bd80525891.slice/crio-50f4c5ff6a8b11ded927d2ae7c2c19e24426c5ba43aa437d0dd67a599f2fa3f5 WatchSource:0}: Error finding container 50f4c5ff6a8b11ded927d2ae7c2c19e24426c5ba43aa437d0dd67a599f2fa3f5: Status 404 returned error can't find the container with id 50f4c5ff6a8b11ded927d2ae7c2c19e24426c5ba43aa437d0dd67a599f2fa3f5 Jul 23 23:14:03.754352 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.754289 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-2cb84"] Jul 23 23:14:03.779260 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:03.779228 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-78c6465647-5pnmv"] Jul 23 23:14:03.783404 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:03.783377 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a47b482_7b1c_4c44_aa8c_23e5b0d262c6.slice/crio-0f5134896db466395a90e403095a68e6eef53c989109943ff74092c23e85bd04 WatchSource:0}: Error finding container 0f5134896db466395a90e403095a68e6eef53c989109943ff74092c23e85bd04: Status 404 returned error can't find the container with id 0f5134896db466395a90e403095a68e6eef53c989109943ff74092c23e85bd04 Jul 23 23:14:04.259843 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.259744 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4"] Jul 23 23:14:04.273634 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.273085 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9"] Jul 23 23:14:04.273634 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.273272 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.280608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.279207 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9"] Jul 23 23:14:04.280608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.280131 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 23:14:04.280608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.280229 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dmr64\"" Jul 23 23:14:04.280608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.280391 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 23 23:14:04.280608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.280543 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 23:14:04.280998 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.280813 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.290562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.280812 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 23:14:04.290562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.282561 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.290562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.284326 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 23 23:14:04.290562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.284591 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 23 23:14:04.290562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.285015 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 23 23:14:04.290562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.285203 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 23 23:14:04.290562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.290006 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 23:14:04.294577 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.294550 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9"] Jul 23 23:14:04.294787 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.294772 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4"] Jul 23 23:14:04.294915 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.294903 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9"] Jul 23 23:14:04.319055 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.318927 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:04.357245 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.357218 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-2ms2f"] Jul 23 23:14:04.361145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.360950 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-2ms2f" Jul 23 23:14:04.364054 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.363528 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 23:14:04.364054 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.363877 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5vqdj\"" Jul 23 23:14:04.364226 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.364162 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 23:14:04.386819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.386449 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-2ms2f"] Jul 23 23:14:04.410445 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410349 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6a969e4-7ab5-4631-9362-1f46919419cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4\" (UID: \"a6a969e4-7ab5-4631-9362-1f46919419cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4e4551b-89ed-404f-ba05-7a069b356653-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-ca\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410825 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4e4551b-89ed-404f-ba05-7a069b356653-tmp\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410848 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268dr\" (UniqueName: \"kubernetes.io/projected/1cd71d85-ae50-4aad-baf5-1f535b9d4436-kube-api-access-268dr\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-hub\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.410984 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvdv\" (UniqueName: \"kubernetes.io/projected/a6a969e4-7ab5-4631-9362-1f46919419cf-kube-api-access-grvdv\") pod \"managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4\" (UID: \"a6a969e4-7ab5-4631-9362-1f46919419cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.411010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd71d85-ae50-4aad-baf5-1f535b9d4436-additional-service-ca\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.411043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzkx\" (UniqueName: \"kubernetes.io/projected/f4e4551b-89ed-404f-ba05-7a069b356653-kube-api-access-qjzkx\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.411242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.411078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.466757 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.466673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:04.503081 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.503038 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" event={"ID":"893999eb-09a3-46c0-9f3e-8c7d3e008682","Type":"ContainerStarted","Data":"fd9caccb82696b22ec62155e7a7f1497b9e87fd6fbd6e7a87f9e858bb57e9766"} Jul 23 23:14:04.507220 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.506399 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78c6465647-5pnmv" event={"ID":"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6","Type":"ContainerStarted","Data":"aee0849b246d774f125dca0db628a7b77a31aa652406a4b81e1c9140a51b65be"} Jul 23 23:14:04.507220 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.507186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-78c6465647-5pnmv" event={"ID":"7a47b482-7b1c-4c44-aa8c-23e5b0d262c6","Type":"ContainerStarted","Data":"0f5134896db466395a90e403095a68e6eef53c989109943ff74092c23e85bd04"} Jul 23 23:14:04.509308 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.508234 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:04.510155 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.510046 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" event={"ID":"3e6212a9-3d48-44e8-8a66-4f1b79d75910","Type":"ContainerStarted","Data":"495d88fd8710eb5c0d9dc2c97fe824744ac3ed150dd9bb9050cdd1ca7b0e7e64"} Jul 23 23:14:04.511580 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.511536 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:04.511961 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.511938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268dr\" (UniqueName: \"kubernetes.io/projected/1cd71d85-ae50-4aad-baf5-1f535b9d4436-kube-api-access-268dr\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.512047 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.511995 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-hub\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.512047 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvdv\" (UniqueName: \"kubernetes.io/projected/a6a969e4-7ab5-4631-9362-1f46919419cf-kube-api-access-grvdv\") pod \"managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4\" (UID: \"a6a969e4-7ab5-4631-9362-1f46919419cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.512150 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512056 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd71d85-ae50-4aad-baf5-1f535b9d4436-additional-service-ca\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.512150 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512085 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzkx\" (UniqueName: \"kubernetes.io/projected/f4e4551b-89ed-404f-ba05-7a069b356653-kube-api-access-qjzkx\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.512150 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbvkp\" (UniqueName: \"kubernetes.io/projected/aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba-kube-api-access-wbvkp\") pod \"downloads-7c97dc4555-2ms2f\" (UID: \"aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba\") " pod="openshift-console/downloads-7c97dc4555-2ms2f" Jul 23 23:14:04.512295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512148 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.512295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6a969e4-7ab5-4631-9362-1f46919419cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4\" (UID: \"a6a969e4-7ab5-4631-9362-1f46919419cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.512295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4e4551b-89ed-404f-ba05-7a069b356653-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.512295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-ca\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.512295 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4e4551b-89ed-404f-ba05-7a069b356653-tmp\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.512542 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.512310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.515016 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.514239 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd71d85-ae50-4aad-baf5-1f535b9d4436-additional-service-ca\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.515115 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.515013 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4e4551b-89ed-404f-ba05-7a069b356653-tmp\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.517852 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.517023 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-hub\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.517852 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.517245 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" event={"ID":"272f9641-d504-4a5f-ae89-7d8b725eb91a","Type":"ContainerStarted","Data":"e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53"} Jul 23 23:14:04.517852 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.517282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" event={"ID":"272f9641-d504-4a5f-ae89-7d8b725eb91a","Type":"ContainerStarted","Data":"364fb88bc57892cbbd209dd34fb6907e765954798c0528221ebc00f16c4332e1"} Jul 23 23:14:04.517852 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.517797 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:14:04.518741 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.518703 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.519939 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.519896 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.520404 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.520225 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6a969e4-7ab5-4631-9362-1f46919419cf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4\" (UID: \"a6a969e4-7ab5-4631-9362-1f46919419cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.520614 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.520567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4e4551b-89ed-404f-ba05-7a069b356653-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.521808 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.521756 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" event={"ID":"b4ef8031-4d56-4616-932e-73bd80525891","Type":"ContainerStarted","Data":"50f4c5ff6a8b11ded927d2ae7c2c19e24426c5ba43aa437d0dd67a599f2fa3f5"} Jul 23 23:14:04.525132 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.525069 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8k2pn" event={"ID":"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f","Type":"ContainerStarted","Data":"19b52eb43a8d530362137feacbbf1466aed1d4eb784db4064a84f404cae6d66a"} Jul 23 23:14:04.525566 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.525544 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1cd71d85-ae50-4aad-baf5-1f535b9d4436-ca\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.527918 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.527580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzkx\" (UniqueName: \"kubernetes.io/projected/f4e4551b-89ed-404f-ba05-7a069b356653-kube-api-access-qjzkx\") pod \"klusterlet-addon-workmgr-6d987746cf-p4vb9\" (UID: \"f4e4551b-89ed-404f-ba05-7a069b356653\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.527918 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.527601 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvdv\" (UniqueName: \"kubernetes.io/projected/a6a969e4-7ab5-4631-9362-1f46919419cf-kube-api-access-grvdv\") pod \"managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4\" (UID: \"a6a969e4-7ab5-4631-9362-1f46919419cf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.527918 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.527879 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-268dr\" (UniqueName: \"kubernetes.io/projected/1cd71d85-ae50-4aad-baf5-1f535b9d4436-kube-api-access-268dr\") pod \"cluster-proxy-proxy-agent-757c5d8f5c-g7rt9\" (UID: \"1cd71d85-ae50-4aad-baf5-1f535b9d4436\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.531119 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.530888 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-78c6465647-5pnmv" podStartSLOduration=35.530870811 podStartE2EDuration="35.530870811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:04.528773353 +0000 UTC m=+68.057050522" watchObservedRunningTime="2026-07-23 23:14:04.530870811 +0000 UTC m=+68.059147980" Jul 23 23:14:04.552814 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.552710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:04.615492 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.612813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbvkp\" (UniqueName: \"kubernetes.io/projected/aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba-kube-api-access-wbvkp\") pod \"downloads-7c97dc4555-2ms2f\" (UID: \"aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba\") " pod="openshift-console/downloads-7c97dc4555-2ms2f" Jul 23 23:14:04.626318 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.626072 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" Jul 23 23:14:04.635628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.635575 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbvkp\" (UniqueName: \"kubernetes.io/projected/aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba-kube-api-access-wbvkp\") pod \"downloads-7c97dc4555-2ms2f\" (UID: \"aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba\") " pod="openshift-console/downloads-7c97dc4555-2ms2f" Jul 23 23:14:04.637324 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.637143 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" Jul 23 23:14:04.655942 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.655917 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:04.659956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.659779 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:04.675503 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.675444 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-2ms2f" Jul 23 23:14:04.760445 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.760336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:04.856433 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:04.856325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:05.136566 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.136508 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" podStartSLOduration=68.136443322 podStartE2EDuration="1m8.136443322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:04.572374514 +0000 UTC m=+68.100651682" watchObservedRunningTime="2026-07-23 23:14:05.136443322 +0000 UTC m=+68.664720493" Jul 23 23:14:05.137363 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.137338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9"] Jul 23 23:14:05.144135 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:05.144105 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e4551b_89ed_404f_ba05_7a069b356653.slice/crio-920337b5d805c4bb32a2fe65c837dc404c91d2996274ae2f24c4d98a34473837 WatchSource:0}: Error finding container 920337b5d805c4bb32a2fe65c837dc404c91d2996274ae2f24c4d98a34473837: Status 404 returned error can't find the container with id 920337b5d805c4bb32a2fe65c837dc404c91d2996274ae2f24c4d98a34473837 Jul 23 23:14:05.175968 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.175734 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9"] Jul 23 23:14:05.183009 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:05.182973 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cd71d85_ae50_4aad_baf5_1f535b9d4436.slice/crio-ddf5bd706630c6b582070f447ff48a88e99dbcb24724377ad2fa2feab2573237 WatchSource:0}: Error finding container ddf5bd706630c6b582070f447ff48a88e99dbcb24724377ad2fa2feab2573237: Status 404 returned error can't find the container with id ddf5bd706630c6b582070f447ff48a88e99dbcb24724377ad2fa2feab2573237 Jul 23 23:14:05.197108 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.197053 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4"] Jul 23 23:14:05.364481 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.363303 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-2ms2f"] Jul 23 23:14:05.533259 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.533137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" event={"ID":"f4e4551b-89ed-404f-ba05-7a069b356653","Type":"ContainerStarted","Data":"920337b5d805c4bb32a2fe65c837dc404c91d2996274ae2f24c4d98a34473837"} Jul 23 23:14:05.535775 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.535692 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" event={"ID":"1cd71d85-ae50-4aad-baf5-1f535b9d4436","Type":"ContainerStarted","Data":"ddf5bd706630c6b582070f447ff48a88e99dbcb24724377ad2fa2feab2573237"} Jul 23 23:14:05.542865 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.542834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xdbwg" event={"ID":"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92","Type":"ContainerStarted","Data":"c9b9ef8e65b79a0290da5ea1677c7783508731e7972c40a8ea690dcd82d33974"} Jul 23 23:14:05.542984 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.542890 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xdbwg" event={"ID":"7933aa4d-a215-4a78-bcfd-ec12e3d6cf92","Type":"ContainerStarted","Data":"10f18e2ed363c48be9f26c8cf2777ecd4f8114a16c104e40a061ba7fe3100524"} Jul 23 23:14:05.542984 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.542920 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:05.544905 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.544857 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-78c6465647-5pnmv" Jul 23 23:14:05.559649 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.559370 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xdbwg" podStartSLOduration=66.709991089 podStartE2EDuration="1m8.559348332s" totalImagesPullingTime="1.849357243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:05.557341785 +0000 UTC m=+69.085618953" watchObservedRunningTime="2026-07-23 23:14:05.559348332 +0000 UTC m=+69.087625500" Jul 23 23:14:05.681761 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.681696 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:05.781438 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:05.781271 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:06.547928 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:06.547870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" event={"ID":"a6a969e4-7ab5-4631-9362-1f46919419cf","Type":"ContainerStarted","Data":"11da8725f5e95cf7327861652eaf2d12153d104d1fb5014f832e0f12a6efa9db"} Jul 23 23:14:06.549498 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:06.549442 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-2ms2f" event={"ID":"aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba","Type":"ContainerStarted","Data":"537c20d6a329ee0488bcd4e927592c6f3a77b153edbfddf9131ce593f4b1eeaf"} Jul 23 23:14:08.907605 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:08.907505 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:09.001105 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:09.001070 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:09.105219 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:09.105124 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:09.154691 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:09.154614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:09.843910 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:09.843845 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:09.907534 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:09.907477 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:10.586728 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:10.586684 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:10.679958 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:10.679922 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:10.902065 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:10.901985 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.007820 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.007741 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.067400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.067314 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.304004 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.303836 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.420030 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.419995 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.518978 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.518935 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.617214 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.617122 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.669991 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.669949 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.827091 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.827059 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:11.928270 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:11.928192 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:12.238394 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:12.238288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:12.313292 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:12.313254 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:13.025504 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:13.025470 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:13.081085 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:13.081051 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:13.431802 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:13.431758 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:13.532086 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:13.532052 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:13.633336 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:13.633303 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:13.736962 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:13.736931 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:14.332192 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:14.332158 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:14.433821 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:14.433786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:16.148117 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:16.148076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:16.734010 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:16.733962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:16.835987 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:16.835952 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:17.152385 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:17.152349 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:17.253766 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:17.253729 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:18.514081 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.514030 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:18.596391 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.596357 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:18.606957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.606920 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" event={"ID":"f4e4551b-89ed-404f-ba05-7a069b356653","Type":"ContainerStarted","Data":"58af78d0ff677b12cd665e43faefc3845d813e9d48ebd28fab0581bf8676f1cf"} Jul 23 23:14:18.607179 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.607161 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:18.608987 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.608943 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" Jul 23 23:14:18.610504 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.610387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" event={"ID":"1cd71d85-ae50-4aad-baf5-1f535b9d4436","Type":"ContainerStarted","Data":"1d0f950a1a42b14519f8761d26d50b2a9d95a71356ed049293d60a21e9b2e30c"} Jul 23 23:14:18.610504 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.610452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" event={"ID":"1cd71d85-ae50-4aad-baf5-1f535b9d4436","Type":"ContainerStarted","Data":"c47bd42117c198dc77104f3b5b47cc71c14de83e5418a551fe9ac80a6aa4c391"} Jul 23 23:14:18.610504 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.610478 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" event={"ID":"1cd71d85-ae50-4aad-baf5-1f535b9d4436","Type":"ContainerStarted","Data":"a9a1f3024f9e2d22e261d61568bc36b95f1673a7b39b43dfeadd35c49792b572"} Jul 23 23:14:18.613521 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.613059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" event={"ID":"893999eb-09a3-46c0-9f3e-8c7d3e008682","Type":"ContainerStarted","Data":"5cc42a4b4403b5b454b6877b28e0133e7ea6f7d6c2b735a5affd0bec52ef93b7"} Jul 23 23:14:18.615578 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.615296 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hc8bv" event={"ID":"d90782c3-1c42-4715-804d-63e1d26969f2","Type":"ContainerStarted","Data":"311aeeae062555f08235d8a9d6394b85d2beca64b598feddb3e2c86981be50f2"} Jul 23 23:14:18.618864 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.618839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" event={"ID":"3e6212a9-3d48-44e8-8a66-4f1b79d75910","Type":"ContainerStarted","Data":"0a5421a9fdcc5d45d0d7322b61e4762166ddc9794aac205e55a51c3dd750f605"} Jul 23 23:14:18.618957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.618890 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" event={"ID":"3e6212a9-3d48-44e8-8a66-4f1b79d75910","Type":"ContainerStarted","Data":"51d92162a03ca82b8c180fe10824729ea057c86a25a281362107525258950345"} Jul 23 23:14:18.620899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.620870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" event={"ID":"b4ef8031-4d56-4616-932e-73bd80525891","Type":"ContainerStarted","Data":"2f9bab644b4ea30ca91576d84670a9a7be1b52c6487df689e25e862085b2c417"} Jul 23 23:14:18.627167 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.627133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8k2pn" event={"ID":"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f","Type":"ContainerStarted","Data":"d6fbd55f80fb5655127367e38f333c889039c8f7018d84895ab04d2b438c97f0"} Jul 23 23:14:18.627282 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.627221 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8k2pn" event={"ID":"fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f","Type":"ContainerStarted","Data":"8a528f23b2ce5f122b073fb11df98b02751300349f30e3e065b42b66a98c41c6"} Jul 23 23:14:18.629755 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.628007 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8k2pn" Jul 23 23:14:18.632701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.632626 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6zhr" event={"ID":"79d0ade1-d285-489d-a7ad-0a1b3afd4943","Type":"ContainerStarted","Data":"cda83422dee84b9a7f2e03a19f6841aaf7ba517d3ba7d8af1809225f7afe4e90"} Jul 23 23:14:18.635081 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.635057 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" event={"ID":"a6a969e4-7ab5-4631-9362-1f46919419cf","Type":"ContainerStarted","Data":"c16c424ec9bca2c0c114021125a72b95f85ebbeb5d9040500e1fc34c0386b476"} Jul 23 23:14:18.644660 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.644577 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d987746cf-p4vb9" podStartSLOduration=1.8250440879999998 podStartE2EDuration="14.6445416s" totalImagesPullingTime="12.819497512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.626650888 +0000 UTC m=+82.154928056" watchObservedRunningTime="2026-07-23 23:14:18.6445416 +0000 UTC m=+82.172818768" Jul 23 23:14:18.644998 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.644944 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-prmzf" podStartSLOduration=36.49160517 podStartE2EDuration="50.64493501s" totalImagesPullingTime="14.15332984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.642909014 +0000 UTC m=+82.171186189" watchObservedRunningTime="2026-07-23 23:14:18.64493501 +0000 UTC m=+82.173212177" Jul 23 23:14:18.657324 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.657263 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hc8bv" podStartSLOduration=66.98670467 podStartE2EDuration="1m21.657240875s" totalImagesPullingTime="14.670536205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.656271554 +0000 UTC m=+82.184548722" watchObservedRunningTime="2026-07-23 23:14:18.657240875 +0000 UTC m=+82.185518044" Jul 23 23:14:18.660482 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.660445 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:18.677277 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.676360 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-757c5d8f5c-g7rt9" podStartSLOduration=1.890763175 podStartE2EDuration="14.676339194s" totalImagesPullingTime="12.785576019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.675333294 +0000 UTC m=+82.203610478" watchObservedRunningTime="2026-07-23 23:14:18.676339194 +0000 UTC m=+82.204616362" Jul 23 23:14:18.706657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.706589 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-dndjn" podStartSLOduration=36.558058024 podStartE2EDuration="50.706567238s" totalImagesPullingTime="14.148509214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.705999414 +0000 UTC m=+82.234276583" watchObservedRunningTime="2026-07-23 23:14:18.706567238 +0000 UTC m=+82.234844474" Jul 23 23:14:18.720533 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.719788 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-2cb84" podStartSLOduration=36.114642233 podStartE2EDuration="49.719769963s" totalImagesPullingTime="13.60512773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.71784135 +0000 UTC m=+82.246118518" watchObservedRunningTime="2026-07-23 23:14:18.719769963 +0000 UTC m=+82.248047110" Jul 23 23:14:18.738813 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.738758 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8k2pn" podStartSLOduration=33.427094358 podStartE2EDuration="47.738738563s" totalImagesPullingTime="14.311644205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.737002084 +0000 UTC m=+82.265279250" watchObservedRunningTime="2026-07-23 23:14:18.738738563 +0000 UTC m=+82.267015734" Jul 23 23:14:18.753072 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.752937 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7dc9db84cb-kcsd4" podStartSLOduration=2.821292979 podStartE2EDuration="14.752836225s" totalImagesPullingTime="11.931543246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:18.75072502 +0000 UTC m=+82.279002188" watchObservedRunningTime="2026-07-23 23:14:18.752836225 +0000 UTC m=+82.281113394" Jul 23 23:14:18.823311 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.823265 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:18.878140 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:18.878100 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:19.245667 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:19.245550 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:20.209264 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:20.209126 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:20.263628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:20.263594 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:23.426173 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:23.426134 2568 patch_prober.go:28] interesting pod/image-registry-5fb7f4dfb5-2cj4v container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 23:14:23.426680 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:23.426208 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 23:14:23.855487 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:23.855443 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:23.909994 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:23.909960 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:25.748837 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:25.748802 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:25.849981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:25.849945 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:26.554259 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:26.554226 2568 patch_prober.go:28] interesting pod/image-registry-5fb7f4dfb5-2cj4v container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 23:14:26.554464 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:26.554289 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 23:14:26.615089 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:26.615052 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:26.694163 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:26.694070 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:27.322925 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:27.322859 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:27.377355 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:27.377319 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:27.551591 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:27.551552 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:29.647166 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.647138 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8k2pn" Jul 23 23:14:29.687790 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.687746 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6zhr" event={"ID":"79d0ade1-d285-489d-a7ad-0a1b3afd4943","Type":"ContainerStarted","Data":"de7bf322c8d34796e931d95e569d40e875362dd0ccdd57382b03e7efa7c6f896"} Jul 23 23:14:29.689526 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.689498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-2ms2f" event={"ID":"aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba","Type":"ContainerStarted","Data":"442eb731eb84741f727a81c49c2e4e54662bcd80fd62dfde41146757d6e01f68"} Jul 23 23:14:29.689680 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.689657 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-2ms2f" Jul 23 23:14:29.690705 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.690678 2568 patch_prober.go:28] interesting pod/downloads-7c97dc4555-2ms2f container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body= Jul 23 23:14:29.690816 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.690744 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-2ms2f" podUID="aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" Jul 23 23:14:29.704306 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.704264 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p6zhr" podStartSLOduration=16.818335206 podStartE2EDuration="47.704250853s" totalImagesPullingTime="30.885915647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:13:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:29.702578976 +0000 UTC m=+93.230856142" watchObservedRunningTime="2026-07-23 23:14:29.704250853 +0000 UTC m=+93.232528041" Jul 23 23:14:29.715264 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:29.715227 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-2ms2f" podStartSLOduration=2.375244908 podStartE2EDuration="25.715215737s" totalImagesPullingTime="23.339970829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:29.714981554 +0000 UTC m=+93.243258722" watchObservedRunningTime="2026-07-23 23:14:29.715215737 +0000 UTC m=+93.243492903" Jul 23 23:14:30.002384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:30.002294 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:30.709474 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:30.709440 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-2ms2f" Jul 23 23:14:31.534144 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:31.534104 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:31.634561 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:31.634521 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:31.736209 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:31.736171 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:31.835538 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:31.835505 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:31.941241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:31.941211 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.012591 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.012545 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.039988 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.039948 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-59zzq"] Jul 23 23:14:32.107283 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.107203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.110016 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.109982 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 23:14:32.110292 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.110268 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 23:14:32.110555 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.110535 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 23:14:32.110877 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.110705 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 23:14:32.110877 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.110575 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-52bjn\"" Jul 23 23:14:32.116549 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.116040 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.174786 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.174757 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02c415ac-93bf-4462-a557-5eb4f6543eba-metrics-client-ca\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175033 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-root\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tqg\" (UniqueName: \"kubernetes.io/projected/02c415ac-93bf-4462-a557-5eb4f6543eba-kube-api-access-47tqg\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175313 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175299 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-sys\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175484 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175471 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-tls\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175617 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-accelerators-collector-config\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175727 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-wtmp\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175834 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175822 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.175951 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.175939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-textfile\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.179679 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.179545 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v"] Jul 23 23:14:32.185652 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.185548 2568 patch_prober.go:28] interesting pod/image-registry-5fb7f4dfb5-2cj4v container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 23:14:32.185652 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.185592 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 23:14:32.210229 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.210205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.277028 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.276990 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.277532 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.277509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02c415ac-93bf-4462-a557-5eb4f6543eba-metrics-client-ca\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.278493 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.278465 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02c415ac-93bf-4462-a557-5eb4f6543eba-metrics-client-ca\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.278643 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.278627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-root\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.278814 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.278653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-root\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.278923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.278776 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tqg\" (UniqueName: \"kubernetes.io/projected/02c415ac-93bf-4462-a557-5eb4f6543eba-kube-api-access-47tqg\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.278923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.278851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-sys\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.278923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.278918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-tls\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.279085 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.278979 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-sys\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.279085 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.279023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-accelerators-collector-config\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.279187 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:14:32.279128 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 23:14:32.279187 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:14:32.279182 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-tls podName:02c415ac-93bf-4462-a557-5eb4f6543eba nodeName:}" failed. No retries permitted until 2026-07-23 23:14:32.779161749 +0000 UTC m=+96.307438911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-tls") pod "node-exporter-59zzq" (UID: "02c415ac-93bf-4462-a557-5eb4f6543eba") : secret "node-exporter-tls" not found Jul 23 23:14:32.279388 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.279372 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-wtmp\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.279551 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.279532 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-wtmp\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.279631 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.279568 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.279685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.279634 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-accelerators-collector-config\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.279919 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.279901 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-textfile\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.280004 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.279606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-textfile\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.289916 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.289867 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.295165 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.295141 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tqg\" (UniqueName: \"kubernetes.io/projected/02c415ac-93bf-4462-a557-5eb4f6543eba-kube-api-access-47tqg\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.371869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.371789 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.463482 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.463448 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.524210 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.524177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:32.786547 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.786459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-tls\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.789198 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.789171 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02c415ac-93bf-4462-a557-5eb4f6543eba-node-exporter-tls\") pod \"node-exporter-59zzq\" (UID: \"02c415ac-93bf-4462-a557-5eb4f6543eba\") " pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:32.926568 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:32.926528 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:33.024363 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:33.024329 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-59zzq" Jul 23 23:14:33.026745 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:33.026707 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:33.064984 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:14:33.064954 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c415ac_93bf_4462_a557_5eb4f6543eba.slice/crio-1ba6978df1180f3b449d808c7007387d9318af7392c92504d403ed84cfa10321 WatchSource:0}: Error finding container 1ba6978df1180f3b449d808c7007387d9318af7392c92504d403ed84cfa10321: Status 404 returned error can't find the container with id 1ba6978df1180f3b449d808c7007387d9318af7392c92504d403ed84cfa10321 Jul 23 23:14:33.096307 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:33.096279 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:33.553396 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:33.553362 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:33.655593 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:33.655559 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:33.710241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:33.710205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59zzq" event={"ID":"02c415ac-93bf-4462-a557-5eb4f6543eba","Type":"ContainerStarted","Data":"1ba6978df1180f3b449d808c7007387d9318af7392c92504d403ed84cfa10321"} Jul 23 23:14:33.711219 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:33.711191 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:34.527795 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:34.527767 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9bxbv" Jul 23 23:14:34.812393 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:34.812363 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:34.913699 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:34.913658 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.011288 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.011213 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.070659 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.070628 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.137147 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.137115 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.255151 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.255114 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.351277 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.351194 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.448837 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.448804 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.635567 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.635473 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:35.727899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.727860 2568 generic.go:356] "Generic (PLEG): container finished" podID="02c415ac-93bf-4462-a557-5eb4f6543eba" containerID="83a64e5b008138dd2eebeded1ef8cc2f4af8b661830fba633119eb9ebb710e0e" exitCode=0 Jul 23 23:14:35.728083 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:35.727955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59zzq" event={"ID":"02c415ac-93bf-4462-a557-5eb4f6543eba","Type":"ContainerDied","Data":"83a64e5b008138dd2eebeded1ef8cc2f4af8b661830fba633119eb9ebb710e0e"} Jul 23 23:14:36.500690 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:36.500655 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:36.555063 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:36.555026 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:36.735401 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:36.735364 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59zzq" event={"ID":"02c415ac-93bf-4462-a557-5eb4f6543eba","Type":"ContainerStarted","Data":"7e5f371acff4f0042c674a504ed959019e5e432e9ce0d3ca81a9f65f2f397fd0"} Jul 23 23:14:36.735951 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:36.735443 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-59zzq" event={"ID":"02c415ac-93bf-4462-a557-5eb4f6543eba","Type":"ContainerStarted","Data":"f1cf2423f400923076e26615a887562da1a992a76840f85d792d7c3806a8ea9d"} Jul 23 23:14:36.755125 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:36.755027 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-59zzq" podStartSLOduration=2.98051832 podStartE2EDuration="4.755006939s" totalImagesPullingTime="1.656291013s" totalInitContainerRuntime="118.197606ms" isStatefulPod=true podCreationTimestamp="2026-07-23 23:14:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:14:36.753543835 +0000 UTC m=+100.281821003" watchObservedRunningTime="2026-07-23 23:14:36.755006939 +0000 UTC m=+100.283284107" Jul 23 23:14:36.806367 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:36.806286 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:36.903685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:36.903654 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:37.314945 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:37.314909 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:37.369064 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:37.369025 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:38.199941 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:38.199903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:38.282532 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:38.282491 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:38.371291 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:38.371247 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:38.743564 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:38.743533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:38.844168 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:38.844122 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:41.577310 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:41.577272 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:41.633386 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:41.633357 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:42.184360 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:42.184332 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:14:45.769467 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:45.769429 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:45.869953 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:45.869918 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:46.288221 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:46.288188 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:46.388309 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:46.388277 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:46.487467 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:46.487392 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:46.773551 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:46.773516 2568 generic.go:356] "Generic (PLEG): container finished" podID="e6f74b68-8640-4e19-b356-b28c92c3f1e4" containerID="1c894f312c73dc602439f81f7d76646e7acfc9b97d022e858f55b90e27b66533" exitCode=0 Jul 23 23:14:46.773911 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:46.773588 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" event={"ID":"e6f74b68-8640-4e19-b356-b28c92c3f1e4","Type":"ContainerDied","Data":"1c894f312c73dc602439f81f7d76646e7acfc9b97d022e858f55b90e27b66533"} Jul 23 23:14:46.773911 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:46.773897 2568 scope.go:117] "RemoveContainer" containerID="1c894f312c73dc602439f81f7d76646e7acfc9b97d022e858f55b90e27b66533" Jul 23 23:14:47.779303 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:47.779269 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rhfnj" event={"ID":"e6f74b68-8640-4e19-b356-b28c92c3f1e4","Type":"ContainerStarted","Data":"84c74a6cd0f9a6575e6602781bed9312c2feb26ad6ae8dda757ebd88e0c6f29e"} Jul 23 23:14:50.644788 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:50.644751 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:50.745330 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:50.745297 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:51.795463 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:51.795368 2568 generic.go:356] "Generic (PLEG): container finished" podID="c7b7d3d4-4175-4601-862b-23b9048ab96b" containerID="52bc9f417e38fae9ba2cdd1298de4aba17eae45127ae23a50415f8ee0604c91b" exitCode=0 Jul 23 23:14:51.795463 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:51.795444 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" event={"ID":"c7b7d3d4-4175-4601-862b-23b9048ab96b","Type":"ContainerDied","Data":"52bc9f417e38fae9ba2cdd1298de4aba17eae45127ae23a50415f8ee0604c91b"} Jul 23 23:14:51.796067 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:51.795843 2568 scope.go:117] "RemoveContainer" containerID="52bc9f417e38fae9ba2cdd1298de4aba17eae45127ae23a50415f8ee0604c91b" Jul 23 23:14:52.801518 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:52.801486 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl" event={"ID":"c7b7d3d4-4175-4601-862b-23b9048ab96b","Type":"ContainerStarted","Data":"94d3b1ae4174eaff9890b93b1a4bdf1c52ca1195009ed3b512a991b19b1fcc81"} Jul 23 23:14:53.862332 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:53.862295 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:53.919698 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:53.919661 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:56.476976 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:56.476934 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:56.582557 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:56.582528 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:56.635230 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:56.635197 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:14:57.208224 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.208191 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerName="registry" containerID="cri-o://e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53" gracePeriod=30 Jul 23 23:14:57.491174 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.491146 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:14:57.591689 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591656 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/272f9641-d504-4a5f-ae89-7d8b725eb91a-ca-trust-extracted\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.591689 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591700 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-certificates\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.591957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591722 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-image-registry-private-configuration\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.591957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-trusted-ca\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.591957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591769 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-bound-sa-token\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.591957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591809 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.591957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591841 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f7d7\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-kube-api-access-4f7d7\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.591957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.591879 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-installation-pull-secrets\") pod \"272f9641-d504-4a5f-ae89-7d8b725eb91a\" (UID: \"272f9641-d504-4a5f-ae89-7d8b725eb91a\") " Jul 23 23:14:57.592266 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.592167 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:14:57.592266 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.592213 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:14:57.594482 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.594443 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-kube-api-access-4f7d7" (OuterVolumeSpecName: "kube-api-access-4f7d7") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "kube-api-access-4f7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:14:57.594623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.594539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:14:57.594623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.594567 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:14:57.594895 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.594874 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:14:57.594986 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.594902 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:14:57.600766 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.600744 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272f9641-d504-4a5f-ae89-7d8b725eb91a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "272f9641-d504-4a5f-ae89-7d8b725eb91a" (UID: "272f9641-d504-4a5f-ae89-7d8b725eb91a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 23:14:57.693037 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693003 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/272f9641-d504-4a5f-ae89-7d8b725eb91a-ca-trust-extracted\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.693037 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693034 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-certificates\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.693037 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693044 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-image-registry-private-configuration\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.693218 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693054 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/272f9641-d504-4a5f-ae89-7d8b725eb91a-trusted-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.693218 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693063 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-bound-sa-token\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.693218 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693071 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-registry-tls\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.693218 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693078 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f7d7\" (UniqueName: \"kubernetes.io/projected/272f9641-d504-4a5f-ae89-7d8b725eb91a-kube-api-access-4f7d7\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.693218 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.693087 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/272f9641-d504-4a5f-ae89-7d8b725eb91a-installation-pull-secrets\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:14:57.821765 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.821732 2568 generic.go:356] "Generic (PLEG): container finished" podID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerID="e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53" exitCode=0 Jul 23 23:14:57.821888 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.821784 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" Jul 23 23:14:57.821888 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.821811 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" event={"ID":"272f9641-d504-4a5f-ae89-7d8b725eb91a","Type":"ContainerDied","Data":"e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53"} Jul 23 23:14:57.821888 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.821864 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v" event={"ID":"272f9641-d504-4a5f-ae89-7d8b725eb91a","Type":"ContainerDied","Data":"364fb88bc57892cbbd209dd34fb6907e765954798c0528221ebc00f16c4332e1"} Jul 23 23:14:57.821888 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.821885 2568 scope.go:117] "RemoveContainer" containerID="e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53" Jul 23 23:14:57.832795 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.832776 2568 scope.go:117] "RemoveContainer" containerID="e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53" Jul 23 23:14:57.833067 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:14:57.833047 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53\": container with ID starting with e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53 not found: ID does not exist" containerID="e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53" Jul 23 23:14:57.833119 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.833075 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53"} err="failed to get container status \"e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53\": rpc error: code = NotFound desc = could not find container \"e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53\": container with ID starting with e45daa6b7e49d0c891f9e142c509f8df25f70239dd36471726c6c5fcc0ffaf53 not found: ID does not exist" Jul 23 23:14:57.843712 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.843692 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v"] Jul 23 23:14:57.851188 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:57.849699 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fb7f4dfb5-2cj4v"] Jul 23 23:14:59.165484 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:14:59.165452 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" path="/var/lib/kubelet/pods/272f9641-d504-4a5f-ae89-7d8b725eb91a/volumes" Jul 23 23:15:01.839668 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:01.839635 2568 generic.go:356] "Generic (PLEG): container finished" podID="888f0508-837a-4dd8-bd36-e5e0c752d917" containerID="936cf4dc8d75091b83e437a6cb131d9e46e2f4b53afb30f2cf417fa70f468616" exitCode=0 Jul 23 23:15:01.840016 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:01.839708 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f9q5n" event={"ID":"888f0508-837a-4dd8-bd36-e5e0c752d917","Type":"ContainerDied","Data":"936cf4dc8d75091b83e437a6cb131d9e46e2f4b53afb30f2cf417fa70f468616"} Jul 23 23:15:01.840016 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:01.840012 2568 scope.go:117] "RemoveContainer" containerID="936cf4dc8d75091b83e437a6cb131d9e46e2f4b53afb30f2cf417fa70f468616" Jul 23 23:15:02.845948 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:02.845916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-f9q5n" event={"ID":"888f0508-837a-4dd8-bd36-e5e0c752d917","Type":"ContainerStarted","Data":"324a6ff09d1f0a9d6c65fce6df930b7f1adf3ebe5d9d06e17ed6ccdf01582624"} Jul 23 23:15:06.646177 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:06.646127 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:06.754693 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:06.754653 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:08.856034 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:08.855995 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:08.907608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:08.907558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:48.227798 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:48.227762 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:48.316710 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:48.316647 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:52.521253 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:52.521151 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:52.622389 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:52.622331 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:52.835561 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:52.835522 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:52.937026 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:52.936989 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:53.703581 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:53.703535 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:15:53.755869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:15:53.755832 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:00.113741 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:00.113708 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:00.161111 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:00.161088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:03.715751 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:03.715703 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:03.768859 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:03.768829 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:03.823081 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:03.823051 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:08.868224 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:08.868137 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:08.968581 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:08.968546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:26.013471 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:26.013432 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:32.588239 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:32.588198 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:32.687620 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:32.687583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:32.863154 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:32.863069 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:32.934280 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:32.934254 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:43.563288 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:43.563251 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:54.212640 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:54.212604 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:54.996691 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:54.996656 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:57.789879 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:57.789842 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:57.845125 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:57.845094 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:16:59.973057 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:16:59.973017 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:00.066386 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:00.066347 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:00.226590 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:00.226512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:00.358131 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:00.358097 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:01.996201 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:01.996168 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:02.387709 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:02.387673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:04.785335 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:04.785295 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:04.838209 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:04.838168 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:10.981683 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:10.981644 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:12.173749 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:12.173713 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:12.233681 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:12.233653 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:12.524653 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:12.524574 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:12.620670 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:12.620634 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:21.001374 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:21.001287 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:21.054948 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:21.054917 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:22.827130 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:22.827096 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:27.065231 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:27.065190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:27.138006 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:27.137966 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:39.026169 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.026135 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:39.135973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.135936 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:39.637691 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.637657 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:39.832761 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.832724 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:39.936644 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.936558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:39.971888 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.971862 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g"] Jul 23 23:17:39.972194 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.972181 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerName="registry" Jul 23 23:17:39.972237 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.972196 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerName="registry" Jul 23 23:17:39.972270 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.972261 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="272f9641-d504-4a5f-ae89-7d8b725eb91a" containerName="registry" Jul 23 23:17:39.975195 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.975181 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:39.977286 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.977266 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 23:17:39.977398 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.977305 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 23:17:39.977398 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.977302 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 23:17:39.977398 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.977324 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 23:17:39.977597 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.977327 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bpjbc\"" Jul 23 23:17:39.986947 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:39.986927 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g"] Jul 23 23:17:40.010807 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.010782 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7v4j\" (UniqueName: \"kubernetes.io/projected/6625b413-418a-45b5-b19c-6ee7b881f9a0-kube-api-access-h7v4j\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.010924 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.010822 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6625b413-418a-45b5-b19c-6ee7b881f9a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.010990 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.010928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6625b413-418a-45b5-b19c-6ee7b881f9a0-tmp\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.010990 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.010982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6625b413-418a-45b5-b19c-6ee7b881f9a0-manifests\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.011095 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.011029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6625b413-418a-45b5-b19c-6ee7b881f9a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.112056 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.112018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7v4j\" (UniqueName: \"kubernetes.io/projected/6625b413-418a-45b5-b19c-6ee7b881f9a0-kube-api-access-h7v4j\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.112542 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.112066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6625b413-418a-45b5-b19c-6ee7b881f9a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.112542 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.112116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6625b413-418a-45b5-b19c-6ee7b881f9a0-tmp\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.112542 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.112155 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6625b413-418a-45b5-b19c-6ee7b881f9a0-manifests\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.112542 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.112191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6625b413-418a-45b5-b19c-6ee7b881f9a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.112790 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.112635 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6625b413-418a-45b5-b19c-6ee7b881f9a0-manifests\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.114614 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.114590 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6625b413-418a-45b5-b19c-6ee7b881f9a0-tmp\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.114752 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.114662 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6625b413-418a-45b5-b19c-6ee7b881f9a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.114752 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.114739 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6625b413-418a-45b5-b19c-6ee7b881f9a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.119256 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.119227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7v4j\" (UniqueName: \"kubernetes.io/projected/6625b413-418a-45b5-b19c-6ee7b881f9a0-kube-api-access-h7v4j\") pod \"opendatahub-operator-controller-manager-5fd4b8775f-rdv7g\" (UID: \"6625b413-418a-45b5-b19c-6ee7b881f9a0\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.160624 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.160599 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:40.261659 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.261590 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:40.285701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.285679 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:40.361795 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.361767 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:40.420610 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.420588 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g"] Jul 23 23:17:40.422804 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:17:40.422775 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6625b413_418a_45b5_b19c_6ee7b881f9a0.slice/crio-d0202560bfade7287143eadd8baf3623cf395e6c927884afac22b38abf85d651 WatchSource:0}: Error finding container d0202560bfade7287143eadd8baf3623cf395e6c927884afac22b38abf85d651: Status 404 returned error can't find the container with id d0202560bfade7287143eadd8baf3623cf395e6c927884afac22b38abf85d651 Jul 23 23:17:40.436826 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.436804 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" event={"ID":"6625b413-418a-45b5-b19c-6ee7b881f9a0","Type":"ContainerStarted","Data":"d0202560bfade7287143eadd8baf3623cf395e6c927884afac22b38abf85d651"} Jul 23 23:17:40.452147 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.452123 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:40.513553 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.513495 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:40.616606 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:40.616574 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:44.455149 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:44.455111 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" event={"ID":"6625b413-418a-45b5-b19c-6ee7b881f9a0","Type":"ContainerStarted","Data":"45bb397e5dbf57c3b578a1b2c7e1166579866b40795c7c68a882847b35bd88af"} Jul 23 23:17:45.460985 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:45.460952 2568 generic.go:356] "Generic (PLEG): container finished" podID="6625b413-418a-45b5-b19c-6ee7b881f9a0" containerID="45bb397e5dbf57c3b578a1b2c7e1166579866b40795c7c68a882847b35bd88af" exitCode=0 Jul 23 23:17:45.461338 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:45.461001 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" event={"ID":"6625b413-418a-45b5-b19c-6ee7b881f9a0","Type":"ContainerDied","Data":"45bb397e5dbf57c3b578a1b2c7e1166579866b40795c7c68a882847b35bd88af"} Jul 23 23:17:46.466967 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:46.466935 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" event={"ID":"6625b413-418a-45b5-b19c-6ee7b881f9a0","Type":"ContainerStarted","Data":"4ab622a0687721a65e18941f21d52a00fd0e09a1f0d9fd0a06b4e3912d1b08c5"} Jul 23 23:17:46.467345 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:46.467032 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:46.485576 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:46.485522 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" podStartSLOduration=2.866120064 podStartE2EDuration="7.485508543s" totalImagesPullingTime="3.564928074s" totalInitContainerRuntime="1.054460405s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:17:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:17:46.483632633 +0000 UTC m=+290.011909812" watchObservedRunningTime="2026-07-23 23:17:46.485508543 +0000 UTC m=+290.013785710" Jul 23 23:17:52.043014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:52.042973 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:52.150656 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:52.150606 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:56.898752 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:56.898724 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:17:56.913456 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:56.913431 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:17:56.914763 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:56.914743 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:17:56.918011 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:56.917996 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 23:17:56.921882 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:56.921864 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:17:57.473242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:57.473211 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fd4b8775f-rdv7g" Jul 23 23:17:59.053448 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.053389 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:17:59.598787 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.598753 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8b54995f6-wzbds"] Jul 23 23:17:59.601981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.601966 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.603960 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.603939 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 23:17:59.604237 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.604216 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 23:17:59.604405 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.604386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kqgw6\"" Jul 23 23:17:59.612666 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.612646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8b54995f6-wzbds"] Jul 23 23:17:59.671953 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.671920 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc357fcf-7057-4f2a-95f2-48e85205127e-tmp\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.672076 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.671964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc357fcf-7057-4f2a-95f2-48e85205127e-tls-certs\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.672076 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.672042 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvvq\" (UniqueName: \"kubernetes.io/projected/fc357fcf-7057-4f2a-95f2-48e85205127e-kube-api-access-zvvvq\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.772859 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.772829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvvq\" (UniqueName: \"kubernetes.io/projected/fc357fcf-7057-4f2a-95f2-48e85205127e-kube-api-access-zvvvq\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.773060 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.772873 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc357fcf-7057-4f2a-95f2-48e85205127e-tmp\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.773060 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.772894 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc357fcf-7057-4f2a-95f2-48e85205127e-tls-certs\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.775104 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.775082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc357fcf-7057-4f2a-95f2-48e85205127e-tmp\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.775316 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.775299 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc357fcf-7057-4f2a-95f2-48e85205127e-tls-certs\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.786211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.786188 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvvq\" (UniqueName: \"kubernetes.io/projected/fc357fcf-7057-4f2a-95f2-48e85205127e-kube-api-access-zvvvq\") pod \"kube-auth-proxy-8b54995f6-wzbds\" (UID: \"fc357fcf-7057-4f2a-95f2-48e85205127e\") " pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.911983 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.911917 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" Jul 23 23:17:59.961603 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.961577 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lz99j"] Jul 23 23:17:59.965293 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.965275 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:17:59.967794 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.967753 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7blch\"" Jul 23 23:17:59.968213 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.968200 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 23:17:59.990500 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:17:59.990467 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lz99j"] Jul 23 23:18:00.055604 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.055577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8b54995f6-wzbds"] Jul 23 23:18:00.057711 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:18:00.057682 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc357fcf_7057_4f2a_95f2_48e85205127e.slice/crio-b79e7abf88ec047bc61e8c9da4fea7b3d645b4a549fe8809ad2100dcc8a70af4 WatchSource:0}: Error finding container b79e7abf88ec047bc61e8c9da4fea7b3d645b4a549fe8809ad2100dcc8a70af4: Status 404 returned error can't find the container with id b79e7abf88ec047bc61e8c9da4fea7b3d645b4a549fe8809ad2100dcc8a70af4 Jul 23 23:18:00.059433 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.059392 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 23:18:00.075277 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.075253 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5nj\" (UniqueName: \"kubernetes.io/projected/b2a54924-975a-4580-866d-7f3dcf714b3f-kube-api-access-jt5nj\") pod \"odh-model-controller-7998c67dc4-lz99j\" (UID: \"b2a54924-975a-4580-866d-7f3dcf714b3f\") " pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.075374 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.075330 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2a54924-975a-4580-866d-7f3dcf714b3f-cert\") pod \"odh-model-controller-7998c67dc4-lz99j\" (UID: \"b2a54924-975a-4580-866d-7f3dcf714b3f\") " pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.176053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.175981 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2a54924-975a-4580-866d-7f3dcf714b3f-cert\") pod \"odh-model-controller-7998c67dc4-lz99j\" (UID: \"b2a54924-975a-4580-866d-7f3dcf714b3f\") " pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.176053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.176023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5nj\" (UniqueName: \"kubernetes.io/projected/b2a54924-975a-4580-866d-7f3dcf714b3f-kube-api-access-jt5nj\") pod \"odh-model-controller-7998c67dc4-lz99j\" (UID: \"b2a54924-975a-4580-866d-7f3dcf714b3f\") " pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.176263 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:18:00.176140 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 23:18:00.176263 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:18:00.176241 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a54924-975a-4580-866d-7f3dcf714b3f-cert podName:b2a54924-975a-4580-866d-7f3dcf714b3f nodeName:}" failed. No retries permitted until 2026-07-23 23:18:00.676219464 +0000 UTC m=+304.204496610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b2a54924-975a-4580-866d-7f3dcf714b3f-cert") pod "odh-model-controller-7998c67dc4-lz99j" (UID: "b2a54924-975a-4580-866d-7f3dcf714b3f") : secret "odh-model-controller-webhook-cert" not found Jul 23 23:18:00.183911 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.183889 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5nj\" (UniqueName: \"kubernetes.io/projected/b2a54924-975a-4580-866d-7f3dcf714b3f-kube-api-access-jt5nj\") pod \"odh-model-controller-7998c67dc4-lz99j\" (UID: \"b2a54924-975a-4580-866d-7f3dcf714b3f\") " pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.528034 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.527945 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" event={"ID":"fc357fcf-7057-4f2a-95f2-48e85205127e","Type":"ContainerStarted","Data":"b79e7abf88ec047bc61e8c9da4fea7b3d645b4a549fe8809ad2100dcc8a70af4"} Jul 23 23:18:00.650351 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.650295 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:00.679917 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.679889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2a54924-975a-4580-866d-7f3dcf714b3f-cert\") pod \"odh-model-controller-7998c67dc4-lz99j\" (UID: \"b2a54924-975a-4580-866d-7f3dcf714b3f\") " pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.683073 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.683007 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2a54924-975a-4580-866d-7f3dcf714b3f-cert\") pod \"odh-model-controller-7998c67dc4-lz99j\" (UID: \"b2a54924-975a-4580-866d-7f3dcf714b3f\") " pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.722804 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.722775 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-74w8s"] Jul 23 23:18:00.726337 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.726309 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:00.728159 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.728138 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 23:18:00.728279 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.728188 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rrxhh\"" Jul 23 23:18:00.736450 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.736363 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-74w8s"] Jul 23 23:18:00.740550 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.740186 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:00.781357 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.781285 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcnl\" (UniqueName: \"kubernetes.io/projected/5a405e7f-594a-4056-a8b6-2e87ab22a26a-kube-api-access-5lcnl\") pod \"model-serving-api-54d89bd5fc-74w8s\" (UID: \"5a405e7f-594a-4056-a8b6-2e87ab22a26a\") " pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:00.781474 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.781396 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a405e7f-594a-4056-a8b6-2e87ab22a26a-tls-certs\") pod \"model-serving-api-54d89bd5fc-74w8s\" (UID: \"5a405e7f-594a-4056-a8b6-2e87ab22a26a\") " pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:00.881307 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.881272 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:00.883129 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.882709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a405e7f-594a-4056-a8b6-2e87ab22a26a-tls-certs\") pod \"model-serving-api-54d89bd5fc-74w8s\" (UID: \"5a405e7f-594a-4056-a8b6-2e87ab22a26a\") " pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:00.883129 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.882774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcnl\" (UniqueName: \"kubernetes.io/projected/5a405e7f-594a-4056-a8b6-2e87ab22a26a-kube-api-access-5lcnl\") pod \"model-serving-api-54d89bd5fc-74w8s\" (UID: \"5a405e7f-594a-4056-a8b6-2e87ab22a26a\") " pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:00.883129 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:18:00.882876 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 23 23:18:00.883129 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:18:00.882956 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a405e7f-594a-4056-a8b6-2e87ab22a26a-tls-certs podName:5a405e7f-594a-4056-a8b6-2e87ab22a26a nodeName:}" failed. No retries permitted until 2026-07-23 23:18:01.3829349 +0000 UTC m=+304.911212056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5a405e7f-594a-4056-a8b6-2e87ab22a26a-tls-certs") pod "model-serving-api-54d89bd5fc-74w8s" (UID: "5a405e7f-594a-4056-a8b6-2e87ab22a26a") : secret "model-serving-api-tls" not found Jul 23 23:18:00.895183 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.895136 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcnl\" (UniqueName: \"kubernetes.io/projected/5a405e7f-594a-4056-a8b6-2e87ab22a26a-kube-api-access-5lcnl\") pod \"model-serving-api-54d89bd5fc-74w8s\" (UID: \"5a405e7f-594a-4056-a8b6-2e87ab22a26a\") " pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:00.919159 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.918687 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r"] Jul 23 23:18:00.922710 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.922691 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:00.924886 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.924601 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 23:18:00.924886 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.924620 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 23:18:00.924886 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.924670 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 23:18:00.924886 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.924625 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 23:18:00.924886 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.924723 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b887s\"" Jul 23 23:18:00.938824 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.938793 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r"] Jul 23 23:18:00.953040 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.953012 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:00.983686 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.983575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:00.983686 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.983657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d303219-42e0-458d-8e58-33761e748a9b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:00.983686 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.983685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:00.983934 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.983714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:00.983934 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.983746 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs7cg\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-kube-api-access-rs7cg\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:00.983934 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.983816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:00.983934 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:00.983862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.043667 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.043545 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lz99j"] Jul 23 23:18:01.045529 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.045102 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:01.048248 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:18:01.048221 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a54924_975a_4580_866d_7f3dcf714b3f.slice/crio-6ad0036ecb79c5632b63105ca15166064a5423104e9972f45d36cbbbd7b71e06 WatchSource:0}: Error finding container 6ad0036ecb79c5632b63105ca15166064a5423104e9972f45d36cbbbd7b71e06: Status 404 returned error can't find the container with id 6ad0036ecb79c5632b63105ca15166064a5423104e9972f45d36cbbbd7b71e06 Jul 23 23:18:01.084696 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.084654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.085104 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.084753 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d303219-42e0-458d-8e58-33761e748a9b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.085104 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.084804 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.085104 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.084835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.085104 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.084868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs7cg\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-kube-api-access-rs7cg\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.085104 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.084898 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.085104 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.084926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.085583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.085537 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.087481 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.087462 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.087629 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.087610 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.087867 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.087845 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.087956 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.087936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d303219-42e0-458d-8e58-33761e748a9b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.092708 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.092688 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs7cg\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-kube-api-access-rs7cg\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.092855 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.092836 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p4b6r\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.145201 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.145164 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:01.234740 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.234712 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:01.372706 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.372682 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r"] Jul 23 23:18:01.376067 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:18:01.376037 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d303219_42e0_458d_8e58_33761e748a9b.slice/crio-9186ee91314c748e9deea04d7dbcff5bce63aebfdd8c82a08bdc4b3299959274 WatchSource:0}: Error finding container 9186ee91314c748e9deea04d7dbcff5bce63aebfdd8c82a08bdc4b3299959274: Status 404 returned error can't find the container with id 9186ee91314c748e9deea04d7dbcff5bce63aebfdd8c82a08bdc4b3299959274 Jul 23 23:18:01.388178 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.388151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a405e7f-594a-4056-a8b6-2e87ab22a26a-tls-certs\") pod \"model-serving-api-54d89bd5fc-74w8s\" (UID: \"5a405e7f-594a-4056-a8b6-2e87ab22a26a\") " pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:01.390923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.390898 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a405e7f-594a-4056-a8b6-2e87ab22a26a-tls-certs\") pod \"model-serving-api-54d89bd5fc-74w8s\" (UID: \"5a405e7f-594a-4056-a8b6-2e87ab22a26a\") " pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:01.535550 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.535509 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" event={"ID":"b2a54924-975a-4580-866d-7f3dcf714b3f","Type":"ContainerStarted","Data":"6ad0036ecb79c5632b63105ca15166064a5423104e9972f45d36cbbbd7b71e06"} Jul 23 23:18:01.536770 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.536735 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" event={"ID":"4d303219-42e0-458d-8e58-33761e748a9b","Type":"ContainerStarted","Data":"9186ee91314c748e9deea04d7dbcff5bce63aebfdd8c82a08bdc4b3299959274"} Jul 23 23:18:01.639910 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.639876 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:01.817477 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.816901 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-74w8s"] Jul 23 23:18:01.870309 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:01.870275 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:02.554859 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:18:02.554812 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a405e7f_594a_4056_a8b6_2e87ab22a26a.slice/crio-7bf048500b71d2fb73eb690915b0c9d3b363092d7fc028668844655ad4ca409e WatchSource:0}: Error finding container 7bf048500b71d2fb73eb690915b0c9d3b363092d7fc028668844655ad4ca409e: Status 404 returned error can't find the container with id 7bf048500b71d2fb73eb690915b0c9d3b363092d7fc028668844655ad4ca409e Jul 23 23:18:03.556875 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:03.556833 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" event={"ID":"5a405e7f-594a-4056-a8b6-2e87ab22a26a","Type":"ContainerStarted","Data":"7bf048500b71d2fb73eb690915b0c9d3b363092d7fc028668844655ad4ca409e"} Jul 23 23:18:03.558705 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:03.558672 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" event={"ID":"fc357fcf-7057-4f2a-95f2-48e85205127e","Type":"ContainerStarted","Data":"ed60321ed54f90b5fdeb013434ef3f7ee3706fd68d53461adf30862c37b12cab"} Jul 23 23:18:03.573912 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:03.573174 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8b54995f6-wzbds" podStartSLOduration=2.025094325 podStartE2EDuration="4.573153322s" totalImagesPullingTime="2.548058997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:18:03.572573919 +0000 UTC m=+307.100851093" watchObservedRunningTime="2026-07-23 23:18:03.573153322 +0000 UTC m=+307.101430491" Jul 23 23:18:05.629732 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:05.629694 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 23:18:05.629999 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:05.629765 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 23 23:18:06.378826 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.378785 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:06.478530 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.478494 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:06.573428 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.573376 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" event={"ID":"5a405e7f-594a-4056-a8b6-2e87ab22a26a","Type":"ContainerStarted","Data":"a1169a247762fb2a5e8ae167d06b2ea19658ba100c4fcaa422315803a9459834"} Jul 23 23:18:06.573601 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.573462 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:06.575004 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.574976 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" event={"ID":"b2a54924-975a-4580-866d-7f3dcf714b3f","Type":"ContainerStarted","Data":"8bc00cdbeb87150a68d53c3bee57736b757c97343edc4598e4d930151c6eca5a"} Jul 23 23:18:06.575137 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.575075 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:06.576749 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.576722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" event={"ID":"4d303219-42e0-458d-8e58-33761e748a9b","Type":"ContainerStarted","Data":"f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9"} Jul 23 23:18:06.576904 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.576888 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:06.578462 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.578441 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-p4b6r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 23:18:06.578552 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.578485 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" podUID="4d303219-42e0-458d-8e58-33761e748a9b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 23:18:06.591459 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.591386 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" podStartSLOduration=3.481598442 podStartE2EDuration="6.591370042s" totalImagesPullingTime="3.1097716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:18:06.590534757 +0000 UTC m=+310.118811937" watchObservedRunningTime="2026-07-23 23:18:06.591370042 +0000 UTC m=+310.119647210" Jul 23 23:18:06.612826 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.612763 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" podStartSLOduration=2.361644047 podStartE2EDuration="6.61274631s" totalImagesPullingTime="4.251102263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:18:06.612122286 +0000 UTC m=+310.140399454" watchObservedRunningTime="2026-07-23 23:18:06.61274631 +0000 UTC m=+310.141023478" Jul 23 23:18:06.667706 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.667605 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" podStartSLOduration=3.081459314 podStartE2EDuration="7.667586076s" totalImagesPullingTime="4.586126762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:18:06.666646985 +0000 UTC m=+310.194924152" watchObservedRunningTime="2026-07-23 23:18:06.667586076 +0000 UTC m=+310.195863449" Jul 23 23:18:06.744724 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.744692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:06.835326 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.835288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:06.942685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:06.942618 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:07.581964 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:07.581936 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:09.914489 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:09.914447 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:09.960012 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:09.959980 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5"] Jul 23 23:18:09.963543 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:09.963524 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:09.966341 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:09.966320 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 23:18:09.966502 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:09.966446 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-cj8qn\"" Jul 23 23:18:09.976972 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:09.976950 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5"] Jul 23 23:18:09.999775 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:09.999735 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:10.064190 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.064158 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:10.067300 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.067274 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb593bef-e501-44e6-9ffb-7b826a62447b-cert\") pod \"kserve-controller-manager-d6f4b65b8-rv9m5\" (UID: \"eb593bef-e501-44e6-9ffb-7b826a62447b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:10.067430 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.067344 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ctx\" (UniqueName: \"kubernetes.io/projected/eb593bef-e501-44e6-9ffb-7b826a62447b-kube-api-access-74ctx\") pod \"kserve-controller-manager-d6f4b65b8-rv9m5\" (UID: \"eb593bef-e501-44e6-9ffb-7b826a62447b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:10.168200 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.168121 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb593bef-e501-44e6-9ffb-7b826a62447b-cert\") pod \"kserve-controller-manager-d6f4b65b8-rv9m5\" (UID: \"eb593bef-e501-44e6-9ffb-7b826a62447b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:10.168200 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.168180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ctx\" (UniqueName: \"kubernetes.io/projected/eb593bef-e501-44e6-9ffb-7b826a62447b-kube-api-access-74ctx\") pod \"kserve-controller-manager-d6f4b65b8-rv9m5\" (UID: \"eb593bef-e501-44e6-9ffb-7b826a62447b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:10.168401 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:18:10.168268 2568 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 23 23:18:10.168401 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:18:10.168336 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb593bef-e501-44e6-9ffb-7b826a62447b-cert podName:eb593bef-e501-44e6-9ffb-7b826a62447b nodeName:}" failed. No retries permitted until 2026-07-23 23:18:10.668318013 +0000 UTC m=+314.196595158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/eb593bef-e501-44e6-9ffb-7b826a62447b-cert") pod "kserve-controller-manager-d6f4b65b8-rv9m5" (UID: "eb593bef-e501-44e6-9ffb-7b826a62447b") : secret "kserve-webhook-server-cert" not found Jul 23 23:18:10.180883 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.180860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ctx\" (UniqueName: \"kubernetes.io/projected/eb593bef-e501-44e6-9ffb-7b826a62447b-kube-api-access-74ctx\") pod \"kserve-controller-manager-d6f4b65b8-rv9m5\" (UID: \"eb593bef-e501-44e6-9ffb-7b826a62447b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:10.321769 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.321733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:10.672393 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.672365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb593bef-e501-44e6-9ffb-7b826a62447b-cert\") pod \"kserve-controller-manager-d6f4b65b8-rv9m5\" (UID: \"eb593bef-e501-44e6-9ffb-7b826a62447b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:10.674818 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.674796 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb593bef-e501-44e6-9ffb-7b826a62447b-cert\") pod \"kserve-controller-manager-d6f4b65b8-rv9m5\" (UID: \"eb593bef-e501-44e6-9ffb-7b826a62447b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:10.873809 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:10.873744 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:11.010173 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:11.010142 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5"] Jul 23 23:18:11.012811 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:18:11.012787 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb593bef_e501_44e6_9ffb_7b826a62447b.slice/crio-ac689ad82af3fa094977ae276093fc40297eda83530bc700c0cbe3ebf2d14025 WatchSource:0}: Error finding container ac689ad82af3fa094977ae276093fc40297eda83530bc700c0cbe3ebf2d14025: Status 404 returned error can't find the container with id ac689ad82af3fa094977ae276093fc40297eda83530bc700c0cbe3ebf2d14025 Jul 23 23:18:11.207807 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:11.207719 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:11.311942 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:11.311897 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:11.385825 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:11.385786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:11.600678 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:11.600646 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" event={"ID":"eb593bef-e501-44e6-9ffb-7b826a62447b","Type":"ContainerStarted","Data":"ac689ad82af3fa094977ae276093fc40297eda83530bc700c0cbe3ebf2d14025"} Jul 23 23:18:12.607778 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.607745 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r"] Jul 23 23:18:12.608157 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.607926 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" podUID="4d303219-42e0-458d-8e58-33761e748a9b" containerName="discovery" containerID="cri-o://f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9" gracePeriod=30 Jul 23 23:18:12.873703 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.873647 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:12.991828 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.991800 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-kubeconfig\") pod \"4d303219-42e0-458d-8e58-33761e748a9b\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " Jul 23 23:18:12.991828 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.991830 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-cacerts\") pod \"4d303219-42e0-458d-8e58-33761e748a9b\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " Jul 23 23:18:12.992053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.991884 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-dns-cert\") pod \"4d303219-42e0-458d-8e58-33761e748a9b\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " Jul 23 23:18:12.992053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.992003 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d303219-42e0-458d-8e58-33761e748a9b-local-certs\") pod \"4d303219-42e0-458d-8e58-33761e748a9b\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " Jul 23 23:18:12.992053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.992049 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-istio-token\") pod \"4d303219-42e0-458d-8e58-33761e748a9b\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " Jul 23 23:18:12.992220 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.992094 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs7cg\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-kube-api-access-rs7cg\") pod \"4d303219-42e0-458d-8e58-33761e748a9b\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " Jul 23 23:18:12.992220 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.992146 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-ca-configmap\") pod \"4d303219-42e0-458d-8e58-33761e748a9b\" (UID: \"4d303219-42e0-458d-8e58-33761e748a9b\") " Jul 23 23:18:12.992664 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.992625 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4d303219-42e0-458d-8e58-33761e748a9b" (UID: "4d303219-42e0-458d-8e58-33761e748a9b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:18:12.994324 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.994297 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4d303219-42e0-458d-8e58-33761e748a9b" (UID: "4d303219-42e0-458d-8e58-33761e748a9b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:18:12.994567 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.994548 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d303219-42e0-458d-8e58-33761e748a9b-local-certs" (OuterVolumeSpecName: "local-certs") pod "4d303219-42e0-458d-8e58-33761e748a9b" (UID: "4d303219-42e0-458d-8e58-33761e748a9b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 23:18:12.994701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.994679 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d303219-42e0-458d-8e58-33761e748a9b" (UID: "4d303219-42e0-458d-8e58-33761e748a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:18:12.994759 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.994701 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4d303219-42e0-458d-8e58-33761e748a9b" (UID: "4d303219-42e0-458d-8e58-33761e748a9b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:18:12.994759 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.994702 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-kube-api-access-rs7cg" (OuterVolumeSpecName: "kube-api-access-rs7cg") pod "4d303219-42e0-458d-8e58-33761e748a9b" (UID: "4d303219-42e0-458d-8e58-33761e748a9b"). InnerVolumeSpecName "kube-api-access-rs7cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:18:12.994759 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:12.994723 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-cacerts" (OuterVolumeSpecName: "cacerts") pod "4d303219-42e0-458d-8e58-33761e748a9b" (UID: "4d303219-42e0-458d-8e58-33761e748a9b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:18:13.093212 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.093177 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-ca-configmap\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:18:13.093212 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.093207 2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-kubeconfig\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:18:13.093212 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.093223 2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-cacerts\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:18:13.093518 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.093232 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4d303219-42e0-458d-8e58-33761e748a9b-istio-csr-dns-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:18:13.093518 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.093243 2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d303219-42e0-458d-8e58-33761e748a9b-local-certs\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:18:13.093518 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.093253 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-istio-token\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:18:13.093518 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.093264 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs7cg\" (UniqueName: \"kubernetes.io/projected/4d303219-42e0-458d-8e58-33761e748a9b-kube-api-access-rs7cg\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:18:13.610322 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.610291 2568 generic.go:356] "Generic (PLEG): container finished" podID="4d303219-42e0-458d-8e58-33761e748a9b" containerID="f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9" exitCode=0 Jul 23 23:18:13.610627 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.610345 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" event={"ID":"4d303219-42e0-458d-8e58-33761e748a9b","Type":"ContainerDied","Data":"f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9"} Jul 23 23:18:13.610627 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.610357 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" Jul 23 23:18:13.610627 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.610394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r" event={"ID":"4d303219-42e0-458d-8e58-33761e748a9b","Type":"ContainerDied","Data":"9186ee91314c748e9deea04d7dbcff5bce63aebfdd8c82a08bdc4b3299959274"} Jul 23 23:18:13.610627 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.610438 2568 scope.go:117] "RemoveContainer" containerID="f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9" Jul 23 23:18:13.630082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.630057 2568 scope.go:117] "RemoveContainer" containerID="f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9" Jul 23 23:18:13.630335 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:18:13.630315 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9\": container with ID starting with f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9 not found: ID does not exist" containerID="f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9" Jul 23 23:18:13.630378 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.630343 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9"} err="failed to get container status \"f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9\": rpc error: code = NotFound desc = could not find container \"f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9\": container with ID starting with f663fa3a9415b88155f314ff9df498536e451131eb6c4f63c1b608ab34ea1dc9 not found: ID does not exist" Jul 23 23:18:13.637924 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.637904 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r"] Jul 23 23:18:13.640981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:13.640958 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p4b6r"] Jul 23 23:18:14.617195 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:14.617155 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" event={"ID":"eb593bef-e501-44e6-9ffb-7b826a62447b","Type":"ContainerStarted","Data":"b80dcb4bf98765aa1c191ebdc9c390123af9428257b3232c797d47c13e9719b4"} Jul 23 23:18:14.617572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:14.617245 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:14.633970 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:14.633923 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" podStartSLOduration=3.059066074 podStartE2EDuration="5.633908324s" totalImagesPullingTime="2.57484225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:18:14.632636577 +0000 UTC m=+318.160913748" watchObservedRunningTime="2026-07-23 23:18:14.633908324 +0000 UTC m=+318.162185490" Jul 23 23:18:15.165842 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:15.165811 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d303219-42e0-458d-8e58-33761e748a9b" path="/var/lib/kubelet/pods/4d303219-42e0-458d-8e58-33761e748a9b/volumes" Jul 23 23:18:17.583360 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:17.583332 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lz99j" Jul 23 23:18:17.585035 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:17.585018 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-74w8s" Jul 23 23:18:35.438999 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:35.438961 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:18:45.626305 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:45.626274 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rv9m5" Jul 23 23:18:49.030973 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:18:49.030922 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:00.639655 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.639622 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:00.742839 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.742800 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:00.796063 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.796034 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7s22m"] Jul 23 23:19:00.796371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.796360 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d303219-42e0-458d-8e58-33761e748a9b" containerName="discovery" Jul 23 23:19:00.796431 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.796373 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d303219-42e0-458d-8e58-33761e748a9b" containerName="discovery" Jul 23 23:19:00.796475 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.796445 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d303219-42e0-458d-8e58-33761e748a9b" containerName="discovery" Jul 23 23:19:00.801107 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.801085 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" Jul 23 23:19:00.803116 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.803093 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 23:19:00.803237 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.803099 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 23:19:00.803237 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.803120 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-4ldpx\"" Jul 23 23:19:00.811381 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.811363 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7s22m"] Jul 23 23:19:00.840651 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.840625 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:00.889903 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.889837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v722k\" (UniqueName: \"kubernetes.io/projected/ded5f47b-141c-4f95-8097-6f0c87bfa376-kube-api-access-v722k\") pod \"kuadrant-operator-catalog-7s22m\" (UID: \"ded5f47b-141c-4f95-8097-6f0c87bfa376\") " pod="kuadrant-system/kuadrant-operator-catalog-7s22m" Jul 23 23:19:00.990723 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:00.990698 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v722k\" (UniqueName: \"kubernetes.io/projected/ded5f47b-141c-4f95-8097-6f0c87bfa376-kube-api-access-v722k\") pod \"kuadrant-operator-catalog-7s22m\" (UID: \"ded5f47b-141c-4f95-8097-6f0c87bfa376\") " pod="kuadrant-system/kuadrant-operator-catalog-7s22m" Jul 23 23:19:01.000981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.000950 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v722k\" (UniqueName: \"kubernetes.io/projected/ded5f47b-141c-4f95-8097-6f0c87bfa376-kube-api-access-v722k\") pod \"kuadrant-operator-catalog-7s22m\" (UID: \"ded5f47b-141c-4f95-8097-6f0c87bfa376\") " pod="kuadrant-system/kuadrant-operator-catalog-7s22m" Jul 23 23:19:01.113470 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.113445 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" Jul 23 23:19:01.168847 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.168818 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7s22m"] Jul 23 23:19:01.245656 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.245633 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7s22m"] Jul 23 23:19:01.247767 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:01.247740 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded5f47b_141c_4f95_8097_6f0c87bfa376.slice/crio-bf5ad8cc9ec81c5fb88f023f0d8486a4eb037a6e83e5b57cc3c9fc10ac13c2cc WatchSource:0}: Error finding container bf5ad8cc9ec81c5fb88f023f0d8486a4eb037a6e83e5b57cc3c9fc10ac13c2cc: Status 404 returned error can't find the container with id bf5ad8cc9ec81c5fb88f023f0d8486a4eb037a6e83e5b57cc3c9fc10ac13c2cc Jul 23 23:19:01.371517 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.371487 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fmbzg"] Jul 23 23:19:01.375829 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.375813 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:01.383786 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.383762 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fmbzg"] Jul 23 23:19:01.494765 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.494690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9zw\" (UniqueName: \"kubernetes.io/projected/390e0cea-0644-44e6-9865-2419f2aca459-kube-api-access-wh9zw\") pod \"kuadrant-operator-catalog-fmbzg\" (UID: \"390e0cea-0644-44e6-9865-2419f2aca459\") " pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:01.592166 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.592140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:01.596107 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.596084 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9zw\" (UniqueName: \"kubernetes.io/projected/390e0cea-0644-44e6-9865-2419f2aca459-kube-api-access-wh9zw\") pod \"kuadrant-operator-catalog-fmbzg\" (UID: \"390e0cea-0644-44e6-9865-2419f2aca459\") " pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:01.603789 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.603767 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9zw\" (UniqueName: \"kubernetes.io/projected/390e0cea-0644-44e6-9865-2419f2aca459-kube-api-access-wh9zw\") pod \"kuadrant-operator-catalog-fmbzg\" (UID: \"390e0cea-0644-44e6-9865-2419f2aca459\") " pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:01.685647 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.685618 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:01.812192 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.812165 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:01.822331 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.822309 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" event={"ID":"ded5f47b-141c-4f95-8097-6f0c87bfa376","Type":"ContainerStarted","Data":"bf5ad8cc9ec81c5fb88f023f0d8486a4eb037a6e83e5b57cc3c9fc10ac13c2cc"} Jul 23 23:19:01.824389 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:01.824368 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390e0cea_0644_44e6_9865_2419f2aca459.slice/crio-3becbfee081b863ad7f4001e17e8ab78df5b55d79099e3cbc0e2a7aa19ba80eb WatchSource:0}: Error finding container 3becbfee081b863ad7f4001e17e8ab78df5b55d79099e3cbc0e2a7aa19ba80eb: Status 404 returned error can't find the container with id 3becbfee081b863ad7f4001e17e8ab78df5b55d79099e3cbc0e2a7aa19ba80eb Jul 23 23:19:01.825542 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:01.825523 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fmbzg"] Jul 23 23:19:02.828765 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:02.828732 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" event={"ID":"390e0cea-0644-44e6-9865-2419f2aca459","Type":"ContainerStarted","Data":"3becbfee081b863ad7f4001e17e8ab78df5b55d79099e3cbc0e2a7aa19ba80eb"} Jul 23 23:19:03.834145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:03.834105 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" event={"ID":"390e0cea-0644-44e6-9865-2419f2aca459","Type":"ContainerStarted","Data":"82023a54ec2da653e8d4fdf5c8e51819db8dd9cb43cda67bb4a704b77226f051"} Jul 23 23:19:03.835397 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:03.835375 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" event={"ID":"ded5f47b-141c-4f95-8097-6f0c87bfa376","Type":"ContainerStarted","Data":"16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23"} Jul 23 23:19:03.835528 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:03.835470 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" podUID="ded5f47b-141c-4f95-8097-6f0c87bfa376" containerName="registry-server" containerID="cri-o://16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23" gracePeriod=2 Jul 23 23:19:03.847834 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:03.847772 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" podStartSLOduration=1.3327799009999999 podStartE2EDuration="2.847755999s" totalImagesPullingTime="1.514976098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:03.847617346 +0000 UTC m=+367.375894513" watchObservedRunningTime="2026-07-23 23:19:03.847755999 +0000 UTC m=+367.376033176" Jul 23 23:19:03.862123 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:03.862084 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" podStartSLOduration=1.774050672 podStartE2EDuration="3.862071737s" totalImagesPullingTime="2.088021065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:03.860591192 +0000 UTC m=+367.388868360" watchObservedRunningTime="2026-07-23 23:19:03.862071737 +0000 UTC m=+367.390348949" Jul 23 23:19:04.074926 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.074902 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" Jul 23 23:19:04.223980 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.223906 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v722k\" (UniqueName: \"kubernetes.io/projected/ded5f47b-141c-4f95-8097-6f0c87bfa376-kube-api-access-v722k\") pod \"ded5f47b-141c-4f95-8097-6f0c87bfa376\" (UID: \"ded5f47b-141c-4f95-8097-6f0c87bfa376\") " Jul 23 23:19:04.226006 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.225983 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded5f47b-141c-4f95-8097-6f0c87bfa376-kube-api-access-v722k" (OuterVolumeSpecName: "kube-api-access-v722k") pod "ded5f47b-141c-4f95-8097-6f0c87bfa376" (UID: "ded5f47b-141c-4f95-8097-6f0c87bfa376"). InnerVolumeSpecName "kube-api-access-v722k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:19:04.325314 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.325275 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v722k\" (UniqueName: \"kubernetes.io/projected/ded5f47b-141c-4f95-8097-6f0c87bfa376-kube-api-access-v722k\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:19:04.841578 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.841538 2568 generic.go:356] "Generic (PLEG): container finished" podID="ded5f47b-141c-4f95-8097-6f0c87bfa376" containerID="16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23" exitCode=0 Jul 23 23:19:04.842020 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.841585 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" Jul 23 23:19:04.842020 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.841633 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" event={"ID":"ded5f47b-141c-4f95-8097-6f0c87bfa376","Type":"ContainerDied","Data":"16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23"} Jul 23 23:19:04.842020 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.841699 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7s22m" event={"ID":"ded5f47b-141c-4f95-8097-6f0c87bfa376","Type":"ContainerDied","Data":"bf5ad8cc9ec81c5fb88f023f0d8486a4eb037a6e83e5b57cc3c9fc10ac13c2cc"} Jul 23 23:19:04.842020 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.841731 2568 scope.go:117] "RemoveContainer" containerID="16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23" Jul 23 23:19:04.850328 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.850311 2568 scope.go:117] "RemoveContainer" containerID="16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23" Jul 23 23:19:04.850577 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:04.850559 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23\": container with ID starting with 16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23 not found: ID does not exist" containerID="16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23" Jul 23 23:19:04.850629 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.850587 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23"} err="failed to get container status \"16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23\": rpc error: code = NotFound desc = could not find container \"16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23\": container with ID starting with 16d5534e6a98c3c248f614782bad85691f082f07e24c3091b5763ea3376b6a23 not found: ID does not exist" Jul 23 23:19:04.860933 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.860914 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7s22m"] Jul 23 23:19:04.864070 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:04.864053 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7s22m"] Jul 23 23:19:05.165688 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:05.165617 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded5f47b-141c-4f95-8097-6f0c87bfa376" path="/var/lib/kubelet/pods/ded5f47b-141c-4f95-8097-6f0c87bfa376/volumes" Jul 23 23:19:11.686603 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:11.686571 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:11.686985 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:11.686740 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:11.708042 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:11.708021 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:11.892885 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:11.892862 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-fmbzg" Jul 23 23:19:30.402904 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:30.402861 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:33.032731 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.032697 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp"] Jul 23 23:19:33.033300 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.033044 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded5f47b-141c-4f95-8097-6f0c87bfa376" containerName="registry-server" Jul 23 23:19:33.033300 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.033055 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded5f47b-141c-4f95-8097-6f0c87bfa376" containerName="registry-server" Jul 23 23:19:33.033300 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.033109 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded5f47b-141c-4f95-8097-6f0c87bfa376" containerName="registry-server" Jul 23 23:19:33.046593 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.046564 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp"] Jul 23 23:19:33.046718 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.046655 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:33.048662 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.048634 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8ckc7\"" Jul 23 23:19:33.144131 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.144098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrzt\" (UniqueName: \"kubernetes.io/projected/479b745e-77d4-41ad-8ee8-b78895d46d52-kube-api-access-7mrzt\") pod \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" (UID: \"479b745e-77d4-41ad-8ee8-b78895d46d52\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:33.245331 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.245299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrzt\" (UniqueName: \"kubernetes.io/projected/479b745e-77d4-41ad-8ee8-b78895d46d52-kube-api-access-7mrzt\") pod \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" (UID: \"479b745e-77d4-41ad-8ee8-b78895d46d52\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:33.263018 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.262986 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrzt\" (UniqueName: \"kubernetes.io/projected/479b745e-77d4-41ad-8ee8-b78895d46d52-kube-api-access-7mrzt\") pod \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" (UID: \"479b745e-77d4-41ad-8ee8-b78895d46d52\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:33.358137 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.358093 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:33.522822 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.522791 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp"] Jul 23 23:19:33.525501 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:33.525467 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479b745e_77d4_41ad_8ee8_b78895d46d52.slice/crio-f5121ded1bef75ad409950747ffad7d850eb68f22dd2903295dd86ed348afeca WatchSource:0}: Error finding container f5121ded1bef75ad409950747ffad7d850eb68f22dd2903295dd86ed348afeca: Status 404 returned error can't find the container with id f5121ded1bef75ad409950747ffad7d850eb68f22dd2903295dd86ed348afeca Jul 23 23:19:33.967587 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:33.967541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" event={"ID":"479b745e-77d4-41ad-8ee8-b78895d46d52","Type":"ContainerStarted","Data":"f5121ded1bef75ad409950747ffad7d850eb68f22dd2903295dd86ed348afeca"} Jul 23 23:19:34.013197 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:34.013164 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:34.113807 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:34.113751 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:35.978281 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:35.978246 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" event={"ID":"479b745e-77d4-41ad-8ee8-b78895d46d52","Type":"ContainerStarted","Data":"c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462"} Jul 23 23:19:35.978711 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:35.978377 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:35.993643 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:35.993594 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" podStartSLOduration=1.330002666 podStartE2EDuration="2.993577783s" totalImagesPullingTime="1.663575117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:35.992327176 +0000 UTC m=+399.520604343" watchObservedRunningTime="2026-07-23 23:19:35.993577783 +0000 UTC m=+399.521854951" Jul 23 23:19:38.171340 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.171306 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm"] Jul 23 23:19:38.174974 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.174958 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.177296 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.177280 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m2lst\"" Jul 23 23:19:38.183659 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.183635 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv6xc\" (UniqueName: \"kubernetes.io/projected/e8b270b0-54d8-4a30-8d3b-d8ce02165630-kube-api-access-qv6xc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.183759 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.183678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8b270b0-54d8-4a30-8d3b-d8ce02165630-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.186869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.186846 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm"] Jul 23 23:19:38.284366 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.284336 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv6xc\" (UniqueName: \"kubernetes.io/projected/e8b270b0-54d8-4a30-8d3b-d8ce02165630-kube-api-access-qv6xc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.284366 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.284368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8b270b0-54d8-4a30-8d3b-d8ce02165630-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.284717 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.284698 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8b270b0-54d8-4a30-8d3b-d8ce02165630-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.291800 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.291775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv6xc\" (UniqueName: \"kubernetes.io/projected/e8b270b0-54d8-4a30-8d3b-d8ce02165630-kube-api-access-qv6xc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.484400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.484324 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:38.623633 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.623597 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm"] Jul 23 23:19:38.625126 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.625100 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:38.628552 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:38.628527 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b270b0_54d8_4a30_8d3b_d8ce02165630.slice/crio-88cbdcb166d2de6eac8a5e401f627bf6f0d9495781527f02dfb221d79bc698dd WatchSource:0}: Error finding container 88cbdcb166d2de6eac8a5e401f627bf6f0d9495781527f02dfb221d79bc698dd: Status 404 returned error can't find the container with id 88cbdcb166d2de6eac8a5e401f627bf6f0d9495781527f02dfb221d79bc698dd Jul 23 23:19:38.992205 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:38.992173 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" event={"ID":"e8b270b0-54d8-4a30-8d3b-d8ce02165630","Type":"ContainerStarted","Data":"88cbdcb166d2de6eac8a5e401f627bf6f0d9495781527f02dfb221d79bc698dd"} Jul 23 23:19:39.008366 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:39.008336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:40.471623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:40.471584 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:40.801917 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:40.801844 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q4plc"] Jul 23 23:19:40.805227 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:40.805208 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" Jul 23 23:19:40.807269 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:40.807248 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-lrl5j\"" Jul 23 23:19:40.814428 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:40.814386 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q4plc"] Jul 23 23:19:40.904011 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:40.903980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tx4n\" (UniqueName: \"kubernetes.io/projected/f28034e7-27ea-4020-a0a9-5f7ad5a5fc16-kube-api-access-9tx4n\") pod \"authorino-operator-75cc8d5cb7-q4plc\" (UID: \"f28034e7-27ea-4020-a0a9-5f7ad5a5fc16\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" Jul 23 23:19:41.004732 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:41.004706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tx4n\" (UniqueName: \"kubernetes.io/projected/f28034e7-27ea-4020-a0a9-5f7ad5a5fc16-kube-api-access-9tx4n\") pod \"authorino-operator-75cc8d5cb7-q4plc\" (UID: \"f28034e7-27ea-4020-a0a9-5f7ad5a5fc16\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" Jul 23 23:19:41.013183 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:41.013162 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tx4n\" (UniqueName: \"kubernetes.io/projected/f28034e7-27ea-4020-a0a9-5f7ad5a5fc16-kube-api-access-9tx4n\") pod \"authorino-operator-75cc8d5cb7-q4plc\" (UID: \"f28034e7-27ea-4020-a0a9-5f7ad5a5fc16\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" Jul 23 23:19:41.115860 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:41.115832 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" Jul 23 23:19:41.250508 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:41.250475 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q4plc"] Jul 23 23:19:41.254677 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:41.254646 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28034e7_27ea_4020_a0a9_5f7ad5a5fc16.slice/crio-14e6495542d5c0d3935c25bdfa64a910d6d3da5709d0473fc40cb06843916cce WatchSource:0}: Error finding container 14e6495542d5c0d3935c25bdfa64a910d6d3da5709d0473fc40cb06843916cce: Status 404 returned error can't find the container with id 14e6495542d5c0d3935c25bdfa64a910d6d3da5709d0473fc40cb06843916cce Jul 23 23:19:42.011122 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:42.011080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" event={"ID":"f28034e7-27ea-4020-a0a9-5f7ad5a5fc16","Type":"ContainerStarted","Data":"14e6495542d5c0d3935c25bdfa64a910d6d3da5709d0473fc40cb06843916cce"} Jul 23 23:19:42.023318 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:42.023281 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:42.847844 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:42.847798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:42.943522 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:42.943476 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:44.249102 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.249072 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v"] Jul 23 23:19:44.252859 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.252837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" Jul 23 23:19:44.254840 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.254817 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 23 23:19:44.254944 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.254817 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ttnzk\"" Jul 23 23:19:44.263157 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.263132 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v"] Jul 23 23:19:44.335995 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.335963 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ht4g\" (UniqueName: \"kubernetes.io/projected/d1b4af67-c153-43e0-ba31-aa3a47f5d737-kube-api-access-5ht4g\") pod \"dns-operator-controller-manager-864dd78c4c-bwv2v\" (UID: \"d1b4af67-c153-43e0-ba31-aa3a47f5d737\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" Jul 23 23:19:44.437259 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.437224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ht4g\" (UniqueName: \"kubernetes.io/projected/d1b4af67-c153-43e0-ba31-aa3a47f5d737-kube-api-access-5ht4g\") pod \"dns-operator-controller-manager-864dd78c4c-bwv2v\" (UID: \"d1b4af67-c153-43e0-ba31-aa3a47f5d737\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" Jul 23 23:19:44.448749 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.448710 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ht4g\" (UniqueName: \"kubernetes.io/projected/d1b4af67-c153-43e0-ba31-aa3a47f5d737-kube-api-access-5ht4g\") pod \"dns-operator-controller-manager-864dd78c4c-bwv2v\" (UID: \"d1b4af67-c153-43e0-ba31-aa3a47f5d737\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" Jul 23 23:19:44.565384 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.565363 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" Jul 23 23:19:44.701237 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.701200 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v"] Jul 23 23:19:44.703924 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:44.703899 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b4af67_c153_43e0_ba31_aa3a47f5d737.slice/crio-53a3c256314e8debf07041449176beb80c14971b927a0b8f8f07e1fc398b61e5 WatchSource:0}: Error finding container 53a3c256314e8debf07041449176beb80c14971b927a0b8f8f07e1fc398b61e5: Status 404 returned error can't find the container with id 53a3c256314e8debf07041449176beb80c14971b927a0b8f8f07e1fc398b61e5 Jul 23 23:19:44.937511 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:44.937478 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:45.028161 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.028124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" event={"ID":"f28034e7-27ea-4020-a0a9-5f7ad5a5fc16","Type":"ContainerStarted","Data":"c1b4a32bbb58780120ce008155a384cde2b88a5c6e8f1c840d3c14e240d39fb5"} Jul 23 23:19:45.028342 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.028253 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" Jul 23 23:19:45.029495 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.029469 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" event={"ID":"d1b4af67-c153-43e0-ba31-aa3a47f5d737","Type":"ContainerStarted","Data":"53a3c256314e8debf07041449176beb80c14971b927a0b8f8f07e1fc398b61e5"} Jul 23 23:19:45.031325 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.031303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" event={"ID":"e8b270b0-54d8-4a30-8d3b-d8ce02165630","Type":"ContainerStarted","Data":"5fb1578a614b7e4d933cc5d7de0827cb81d8036f61ecdca73e87db672edf4bed"} Jul 23 23:19:45.031434 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.031362 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:45.044759 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.044718 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" podStartSLOduration=1.791413832 podStartE2EDuration="5.044704978s" totalImagesPullingTime="3.253291146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:45.044046838 +0000 UTC m=+408.572324008" watchObservedRunningTime="2026-07-23 23:19:45.044704978 +0000 UTC m=+408.572982145" Jul 23 23:19:45.059824 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.059782 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" podStartSLOduration=1.21517011 podStartE2EDuration="7.059769612s" totalImagesPullingTime="5.844599502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:45.058999995 +0000 UTC m=+408.587277173" watchObservedRunningTime="2026-07-23 23:19:45.059769612 +0000 UTC m=+408.588046781" Jul 23 23:19:45.853796 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.853756 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:45.909182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:45.909129 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:46.212068 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:46.211989 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:46.314828 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:46.314780 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:46.985354 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:46.985325 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:47.044271 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:47.044185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" event={"ID":"d1b4af67-c153-43e0-ba31-aa3a47f5d737","Type":"ContainerStarted","Data":"15a0b610bed06d3ddc1c47aa931d6fdade4c53c4befbf39016c8b99aedb67961"} Jul 23 23:19:47.044445 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:47.044304 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" Jul 23 23:19:47.061388 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:47.061346 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" podStartSLOduration=0.999421541 podStartE2EDuration="3.061330694s" totalImagesPullingTime="2.061909153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:47.05937255 +0000 UTC m=+410.587649717" watchObservedRunningTime="2026-07-23 23:19:47.061330694 +0000 UTC m=+410.589607891" Jul 23 23:19:47.310156 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:47.309907 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:47.500815 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:47.500772 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:49.607209 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:49.607172 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:49.665680 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:49.665646 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:52.425764 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:52.425730 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:52.483902 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:52.483860 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:56.039158 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:56.039128 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:56.039544 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:56.039231 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q4plc" Jul 23 23:19:56.944068 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:56.944035 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:57.050508 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.050479 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:57.891648 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.891609 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp"] Jul 23 23:19:57.896163 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.896147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:57.910639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.910618 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp"] Jul 23 23:19:57.942688 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.942663 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm"] Jul 23 23:19:57.942874 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.942835 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" containerName="manager" containerID="cri-o://5fb1578a614b7e4d933cc5d7de0827cb81d8036f61ecdca73e87db672edf4bed" gracePeriod=2 Jul 23 23:19:57.951623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.951600 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6nz\" (UniqueName: \"kubernetes.io/projected/89347887-e4bf-4aa7-80a9-76e2885ea32d-kube-api-access-jr6nz\") pod \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" (UID: \"89347887-e4bf-4aa7-80a9-76e2885ea32d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:57.951753 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.951733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89347887-e4bf-4aa7-80a9-76e2885ea32d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" (UID: \"89347887-e4bf-4aa7-80a9-76e2885ea32d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:57.959202 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.959180 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm"] Jul 23 23:19:57.967850 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:57.967814 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e8b270b0-54d8-4a30-8d3b-d8ce02165630\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\": pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:57.973937 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.973915 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp"] Jul 23 23:19:57.974151 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:57.974135 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-jr6nz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" Jul 23 23:19:57.980721 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.980695 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp"] Jul 23 23:19:57.994499 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.994475 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp"] Jul 23 23:19:57.994702 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:57.994661 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" containerName="manager" containerID="cri-o://c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462" gracePeriod=2 Jul 23 23:19:57.996237 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:57.996207 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.005904 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.005880 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46"] Jul 23 23:19:58.006375 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.006357 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" containerName="manager" Jul 23 23:19:58.006539 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.006516 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" containerName="manager" Jul 23 23:19:58.006725 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.006699 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" containerName="manager" Jul 23 23:19:58.010478 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.010435 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"479b745e-77d4-41ad-8ee8-b78895d46d52\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-s4lvp\": pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.010940 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.010902 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp"] Jul 23 23:19:58.011103 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.011026 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.014027 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.013877 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.022821 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.022804 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml"] Jul 23 23:19:58.023148 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.023133 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" containerName="manager" Jul 23 23:19:58.023216 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.023151 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" containerName="manager" Jul 23 23:19:58.023265 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.023241 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" containerName="manager" Jul 23 23:19:58.026110 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.026093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46"] Jul 23 23:19:58.026222 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.026190 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:19:58.040390 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.040367 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml"] Jul 23 23:19:58.050078 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.050046 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.051541 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.051517 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bwv2v" Jul 23 23:19:58.051996 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.051739 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.052440 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.052398 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89347887-e4bf-4aa7-80a9-76e2885ea32d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" (UID: \"89347887-e4bf-4aa7-80a9-76e2885ea32d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.052604 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.052576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6nz\" (UniqueName: \"kubernetes.io/projected/89347887-e4bf-4aa7-80a9-76e2885ea32d-kube-api-access-jr6nz\") pod \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" (UID: \"89347887-e4bf-4aa7-80a9-76e2885ea32d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.053217 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.053184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89347887-e4bf-4aa7-80a9-76e2885ea32d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" (UID: \"89347887-e4bf-4aa7-80a9-76e2885ea32d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.053217 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.053182 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.056075 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.056048 2568 projected.go:196] Error preparing data for projected volume kube-api-access-jr6nz for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-tmwqp" not found Jul 23 23:19:58.056171 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.056104 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89347887-e4bf-4aa7-80a9-76e2885ea32d-kube-api-access-jr6nz podName:89347887-e4bf-4aa7-80a9-76e2885ea32d nodeName:}" failed. No retries permitted until 2026-07-23 23:19:58.556088086 +0000 UTC m=+422.084365231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jr6nz" (UniqueName: "kubernetes.io/projected/89347887-e4bf-4aa7-80a9-76e2885ea32d-kube-api-access-jr6nz") pod "kuadrant-operator-controller-manager-86878c769f-tmwqp" (UID: "89347887-e4bf-4aa7-80a9-76e2885ea32d") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-tmwqp" not found Jul 23 23:19:58.103487 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.103462 2568 generic.go:356] "Generic (PLEG): container finished" podID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" containerID="5fb1578a614b7e4d933cc5d7de0827cb81d8036f61ecdca73e87db672edf4bed" exitCode=0 Jul 23 23:19:58.103583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.103568 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.104604 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.104570 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.106839 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.106809 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.108507 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.108461 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.109919 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.109890 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.110840 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.110820 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.111205 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.111184 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.112498 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.112454 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.113695 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.113667 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.114803 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.114775 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.154145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.154054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8jh\" (UniqueName: \"kubernetes.io/projected/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-kube-api-access-2j8jh\") pod \"kuadrant-operator-controller-manager-86878c769f-vgr46\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.154145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.154119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vgr46\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.154340 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.154268 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfpz\" (UniqueName: \"kubernetes.io/projected/16456c9c-4389-4aa1-bcaa-86f1f911360e-kube-api-access-dgfpz\") pod \"limitador-operator-controller-manager-859d7b78bd-q8bml\" (UID: \"16456c9c-4389-4aa1-bcaa-86f1f911360e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:19:58.244745 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.244713 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:58.254586 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.254566 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89347887-e4bf-4aa7-80a9-76e2885ea32d-extensions-socket-volume\") pod \"89347887-e4bf-4aa7-80a9-76e2885ea32d\" (UID: \"89347887-e4bf-4aa7-80a9-76e2885ea32d\") " Jul 23 23:19:58.254819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.254791 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89347887-e4bf-4aa7-80a9-76e2885ea32d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "89347887-e4bf-4aa7-80a9-76e2885ea32d" (UID: "89347887-e4bf-4aa7-80a9-76e2885ea32d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 23:19:58.254819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.254805 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.254995 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.254805 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfpz\" (UniqueName: \"kubernetes.io/projected/16456c9c-4389-4aa1-bcaa-86f1f911360e-kube-api-access-dgfpz\") pod \"limitador-operator-controller-manager-859d7b78bd-q8bml\" (UID: \"16456c9c-4389-4aa1-bcaa-86f1f911360e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:19:58.255054 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.255000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8jh\" (UniqueName: \"kubernetes.io/projected/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-kube-api-access-2j8jh\") pod \"kuadrant-operator-controller-manager-86878c769f-vgr46\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.255139 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.255115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vgr46\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.255524 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.255264 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr6nz\" (UniqueName: \"kubernetes.io/projected/89347887-e4bf-4aa7-80a9-76e2885ea32d-kube-api-access-jr6nz\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:19:58.255524 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.255289 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/89347887-e4bf-4aa7-80a9-76e2885ea32d-extensions-socket-volume\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:19:58.255677 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.255529 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vgr46\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.256477 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.256451 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.257655 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.257613 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.258016 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.257999 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.258798 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.258772 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.260110 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.260089 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:58.261374 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.261344 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:58.262649 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:58.262621 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:58.277176 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.277153 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8jh\" (UniqueName: \"kubernetes.io/projected/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-kube-api-access-2j8jh\") pod \"kuadrant-operator-controller-manager-86878c769f-vgr46\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.277262 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.277244 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfpz\" (UniqueName: \"kubernetes.io/projected/16456c9c-4389-4aa1-bcaa-86f1f911360e-kube-api-access-dgfpz\") pod \"limitador-operator-controller-manager-859d7b78bd-q8bml\" (UID: \"16456c9c-4389-4aa1-bcaa-86f1f911360e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:19:58.347372 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.347342 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:19:58.355542 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.355520 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv6xc\" (UniqueName: \"kubernetes.io/projected/e8b270b0-54d8-4a30-8d3b-d8ce02165630-kube-api-access-qv6xc\") pod \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " Jul 23 23:19:58.355644 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.355629 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8b270b0-54d8-4a30-8d3b-d8ce02165630-extensions-socket-volume\") pod \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\" (UID: \"e8b270b0-54d8-4a30-8d3b-d8ce02165630\") " Jul 23 23:19:58.355705 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.355668 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mrzt\" (UniqueName: \"kubernetes.io/projected/479b745e-77d4-41ad-8ee8-b78895d46d52-kube-api-access-7mrzt\") pod \"479b745e-77d4-41ad-8ee8-b78895d46d52\" (UID: \"479b745e-77d4-41ad-8ee8-b78895d46d52\") " Jul 23 23:19:58.355912 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.355892 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b270b0-54d8-4a30-8d3b-d8ce02165630-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e8b270b0-54d8-4a30-8d3b-d8ce02165630" (UID: "e8b270b0-54d8-4a30-8d3b-d8ce02165630"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 23:19:58.357423 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.357391 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b270b0-54d8-4a30-8d3b-d8ce02165630-kube-api-access-qv6xc" (OuterVolumeSpecName: "kube-api-access-qv6xc") pod "e8b270b0-54d8-4a30-8d3b-d8ce02165630" (UID: "e8b270b0-54d8-4a30-8d3b-d8ce02165630"). InnerVolumeSpecName "kube-api-access-qv6xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:19:58.357487 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.357457 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479b745e-77d4-41ad-8ee8-b78895d46d52-kube-api-access-7mrzt" (OuterVolumeSpecName: "kube-api-access-7mrzt") pod "479b745e-77d4-41ad-8ee8-b78895d46d52" (UID: "479b745e-77d4-41ad-8ee8-b78895d46d52"). InnerVolumeSpecName "kube-api-access-7mrzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:19:58.373726 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.373709 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:58.380528 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.380508 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:19:58.456811 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.456779 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8b270b0-54d8-4a30-8d3b-d8ce02165630-extensions-socket-volume\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:19:58.456811 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.456814 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrzt\" (UniqueName: \"kubernetes.io/projected/479b745e-77d4-41ad-8ee8-b78895d46d52-kube-api-access-7mrzt\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:19:58.456985 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.456827 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv6xc\" (UniqueName: \"kubernetes.io/projected/e8b270b0-54d8-4a30-8d3b-d8ce02165630-kube-api-access-qv6xc\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:19:58.518202 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.518176 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46"] Jul 23 23:19:58.522029 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:58.522002 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a5885d_cde8_4e3e_8e01_b7950d239bbf.slice/crio-33defbddb9640a62de05798afbce52bd9a69ee855768a1948b0173fd05d57444 WatchSource:0}: Error finding container 33defbddb9640a62de05798afbce52bd9a69ee855768a1948b0173fd05d57444: Status 404 returned error can't find the container with id 33defbddb9640a62de05798afbce52bd9a69ee855768a1948b0173fd05d57444 Jul 23 23:19:58.576036 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:58.576014 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml"] Jul 23 23:19:58.585515 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:19:58.585487 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16456c9c_4389_4aa1_bcaa_86f1f911360e.slice/crio-f4ee6447d306ed192d4876357705ffe1ee280a5a860f468fda671f1483fe623c WatchSource:0}: Error finding container f4ee6447d306ed192d4876357705ffe1ee280a5a860f468fda671f1483fe623c: Status 404 returned error can't find the container with id f4ee6447d306ed192d4876357705ffe1ee280a5a860f468fda671f1483fe623c Jul 23 23:19:59.110471 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.110435 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" event={"ID":"b8a5885d-cde8-4e3e-8e01-b7950d239bbf","Type":"ContainerStarted","Data":"fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce"} Jul 23 23:19:59.110913 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.110503 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:19:59.110913 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.110527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" event={"ID":"b8a5885d-cde8-4e3e-8e01-b7950d239bbf","Type":"ContainerStarted","Data":"33defbddb9640a62de05798afbce52bd9a69ee855768a1948b0173fd05d57444"} Jul 23 23:19:59.111571 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.111547 2568 generic.go:356] "Generic (PLEG): container finished" podID="479b745e-77d4-41ad-8ee8-b78895d46d52" containerID="c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462" exitCode=0 Jul 23 23:19:59.111651 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.111591 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:59.111651 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.111632 2568 scope.go:117] "RemoveContainer" containerID="c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462" Jul 23 23:19:59.112325 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:59.112286 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:59.113006 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.112987 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:59.114561 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.114540 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:59.114661 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.114560 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" event={"ID":"16456c9c-4389-4aa1-bcaa-86f1f911360e","Type":"ContainerStarted","Data":"6542d9003795aa76b0ff4af5bbfce70dcf951a3171e865805fd128a6fe4e63f0"} Jul 23 23:19:59.114661 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.114607 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" event={"ID":"16456c9c-4389-4aa1-bcaa-86f1f911360e","Type":"ContainerStarted","Data":"f4ee6447d306ed192d4876357705ffe1ee280a5a860f468fda671f1483fe623c"} Jul 23 23:19:59.121954 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.121939 2568 scope.go:117] "RemoveContainer" containerID="c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462" Jul 23 23:19:59.122232 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:59.122206 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462\": container with ID starting with c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462 not found: ID does not exist" containerID="c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462" Jul 23 23:19:59.122308 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.122240 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462"} err="failed to get container status \"c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462\": rpc error: code = NotFound desc = could not find container \"c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462\": container with ID starting with c16875b1051327d283a879e97476cbc25806c8f4122b1168730a364bce9ac462 not found: ID does not exist" Jul 23 23:19:59.122308 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.122262 2568 scope.go:117] "RemoveContainer" containerID="5fb1578a614b7e4d933cc5d7de0827cb81d8036f61ecdca73e87db672edf4bed" Jul 23 23:19:59.131392 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:59.131359 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:59.132658 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:59.132631 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:59.133179 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.133138 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" podStartSLOduration=2.133122617 podStartE2EDuration="2.133122617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:59.129778541 +0000 UTC m=+422.658055709" watchObservedRunningTime="2026-07-23 23:19:59.133122617 +0000 UTC m=+422.661399785" Jul 23 23:19:59.133924 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:59.133904 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-ql6fm\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ql6fm" Jul 23 23:19:59.135164 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:59.135144 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-tmwqp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tmwqp" Jul 23 23:19:59.136322 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:19:59.136304 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s4lvp\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s4lvp" Jul 23 23:19:59.165320 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.165290 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="479b745e-77d4-41ad-8ee8-b78895d46d52" path="/var/lib/kubelet/pods/479b745e-77d4-41ad-8ee8-b78895d46d52/volumes" Jul 23 23:19:59.165683 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.165667 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89347887-e4bf-4aa7-80a9-76e2885ea32d" path="/var/lib/kubelet/pods/89347887-e4bf-4aa7-80a9-76e2885ea32d/volumes" Jul 23 23:19:59.165933 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:19:59.165917 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b270b0-54d8-4a30-8d3b-d8ce02165630" path="/var/lib/kubelet/pods/e8b270b0-54d8-4a30-8d3b-d8ce02165630/volumes" Jul 23 23:20:00.121212 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:00.121184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:20:00.686374 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:00.686337 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:00.787923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:00.787894 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:04.999760 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:04.999727 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:05.100348 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:05.100309 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:07.241220 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:07.241171 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:07.303539 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:07.303508 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:10.123797 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:10.123764 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:20:10.140456 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:10.140390 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" podStartSLOduration=13.140376365 podStartE2EDuration="13.140376365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:19:59.156550554 +0000 UTC m=+422.684827721" watchObservedRunningTime="2026-07-23 23:20:10.140376365 +0000 UTC m=+433.668653571" Jul 23 23:20:11.128137 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:11.128101 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:20:15.941188 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:15.941146 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:16.056722 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:16.056692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:18.381662 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:18.381624 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:18.483164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:18.483137 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:19.433058 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:19.433019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:19.488375 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:19.488337 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:22.750628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:22.750592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:22.854319 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:22.854286 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:25.029111 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:25.029074 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:26.393806 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:26.393761 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:26.490109 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:26.490081 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:26.596668 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:26.596635 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:26.718676 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:26.718572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:28.220401 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.220365 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:28.229621 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.229594 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hvtpb"] Jul 23 23:20:28.233028 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.233010 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.234939 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.234918 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rtq24\"" Jul 23 23:20:28.235124 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.235105 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 23 23:20:28.241010 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.240547 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hvtpb"] Jul 23 23:20:28.304866 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.304802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grhd\" (UniqueName: \"kubernetes.io/projected/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-kube-api-access-5grhd\") pod \"limitador-limitador-64c94fd64f-hvtpb\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.304866 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.304869 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-config-file\") pod \"limitador-limitador-64c94fd64f-hvtpb\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.324771 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.324737 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hvtpb"] Jul 23 23:20:28.331999 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.331967 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:28.385962 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.385933 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:28.406024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.406000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grhd\" (UniqueName: \"kubernetes.io/projected/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-kube-api-access-5grhd\") pod \"limitador-limitador-64c94fd64f-hvtpb\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.406116 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.406034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-config-file\") pod \"limitador-limitador-64c94fd64f-hvtpb\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.406600 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.406583 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-config-file\") pod \"limitador-limitador-64c94fd64f-hvtpb\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.412931 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.412906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grhd\" (UniqueName: \"kubernetes.io/projected/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-kube-api-access-5grhd\") pod \"limitador-limitador-64c94fd64f-hvtpb\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.544839 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.544765 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:28.675053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:28.675026 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hvtpb"] Jul 23 23:20:28.677914 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:20:28.677883 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281d1043_3d93_4f4b_bc3f_c474fdb8bd7f.slice/crio-9f9b940ecff9031b0ca87b88575d6aeb4455616db243d75ffc3b87f0b29f0476 WatchSource:0}: Error finding container 9f9b940ecff9031b0ca87b88575d6aeb4455616db243d75ffc3b87f0b29f0476: Status 404 returned error can't find the container with id 9f9b940ecff9031b0ca87b88575d6aeb4455616db243d75ffc3b87f0b29f0476 Jul 23 23:20:29.054109 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:29.054074 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:29.153885 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:29.153849 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:29.258940 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:29.258906 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" event={"ID":"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f","Type":"ContainerStarted","Data":"9f9b940ecff9031b0ca87b88575d6aeb4455616db243d75ffc3b87f0b29f0476"} Jul 23 23:20:32.276980 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:32.276946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" event={"ID":"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f","Type":"ContainerStarted","Data":"afd58bc6821b5e68225de3175ae918a274e9bcac922b487461cefc2503098458"} Jul 23 23:20:32.277382 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:32.277082 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:32.292509 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:32.292464 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" podStartSLOduration=1.5788247640000002 podStartE2EDuration="4.292450045s" totalImagesPullingTime="2.713625281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:20:32.291200406 +0000 UTC m=+455.819477572" watchObservedRunningTime="2026-07-23 23:20:32.292450045 +0000 UTC m=+455.820727211" Jul 23 23:20:35.750525 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:35.750483 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:35.850373 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:35.850335 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:38.206765 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:38.206726 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:38.273994 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:38.273964 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:43.283541 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:43.283512 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:47.967083 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:47.967049 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hvtpb"] Jul 23 23:20:47.967564 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:47.967244 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" podUID="281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" containerName="limitador" containerID="cri-o://afd58bc6821b5e68225de3175ae918a274e9bcac922b487461cefc2503098458" gracePeriod=30 Jul 23 23:20:48.353729 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:48.353698 2568 generic.go:356] "Generic (PLEG): container finished" podID="281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" containerID="afd58bc6821b5e68225de3175ae918a274e9bcac922b487461cefc2503098458" exitCode=0 Jul 23 23:20:48.353865 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:48.353745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" event={"ID":"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f","Type":"ContainerDied","Data":"afd58bc6821b5e68225de3175ae918a274e9bcac922b487461cefc2503098458"} Jul 23 23:20:48.920563 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:48.920541 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:48.977548 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:48.977522 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5grhd\" (UniqueName: \"kubernetes.io/projected/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-kube-api-access-5grhd\") pod \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " Jul 23 23:20:48.977859 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:48.977553 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-config-file\") pod \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\" (UID: \"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f\") " Jul 23 23:20:48.977942 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:48.977917 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-config-file" (OuterVolumeSpecName: "config-file") pod "281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" (UID: "281d1043-3d93-4f4b-bc3f-c474fdb8bd7f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:20:48.979470 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:48.979448 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-kube-api-access-5grhd" (OuterVolumeSpecName: "kube-api-access-5grhd") pod "281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" (UID: "281d1043-3d93-4f4b-bc3f-c474fdb8bd7f"). InnerVolumeSpecName "kube-api-access-5grhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:20:49.078593 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.078564 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5grhd\" (UniqueName: \"kubernetes.io/projected/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-kube-api-access-5grhd\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:20:49.078593 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.078593 2568 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f-config-file\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:20:49.359781 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.359708 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" Jul 23 23:20:49.359781 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.359738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-hvtpb" event={"ID":"281d1043-3d93-4f4b-bc3f-c474fdb8bd7f","Type":"ContainerDied","Data":"9f9b940ecff9031b0ca87b88575d6aeb4455616db243d75ffc3b87f0b29f0476"} Jul 23 23:20:49.359988 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.359789 2568 scope.go:117] "RemoveContainer" containerID="afd58bc6821b5e68225de3175ae918a274e9bcac922b487461cefc2503098458" Jul 23 23:20:49.387837 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.387806 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hvtpb"] Jul 23 23:20:49.397247 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.397221 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-hvtpb"] Jul 23 23:20:49.435935 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.435900 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:49.558747 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.558712 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:49.878690 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.878658 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5"] Jul 23 23:20:49.879209 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.879190 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" containerName="limitador" Jul 23 23:20:49.879279 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.879214 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" containerName="limitador" Jul 23 23:20:49.879352 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.879340 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" containerName="limitador" Jul 23 23:20:49.883903 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.883884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:49.886022 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.886002 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 23 23:20:49.886225 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.886008 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 23:20:49.886316 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.886024 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 23:20:49.886316 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.886059 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-22zn8\"" Jul 23 23:20:49.887604 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.887585 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5"] Jul 23 23:20:49.908704 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.908671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:20:49.985490 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.985462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbdt\" (UniqueName: \"kubernetes.io/projected/1e62999c-0ddd-410b-817a-3b744196df42-kube-api-access-5lbdt\") pod \"postgres-69fb4594cf-sdmv5\" (UID: \"1e62999c-0ddd-410b-817a-3b744196df42\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:49.985816 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:49.985506 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1e62999c-0ddd-410b-817a-3b744196df42-data\") pod \"postgres-69fb4594cf-sdmv5\" (UID: \"1e62999c-0ddd-410b-817a-3b744196df42\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:50.086806 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:50.086768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbdt\" (UniqueName: \"kubernetes.io/projected/1e62999c-0ddd-410b-817a-3b744196df42-kube-api-access-5lbdt\") pod \"postgres-69fb4594cf-sdmv5\" (UID: \"1e62999c-0ddd-410b-817a-3b744196df42\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:50.086989 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:50.086827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1e62999c-0ddd-410b-817a-3b744196df42-data\") pod \"postgres-69fb4594cf-sdmv5\" (UID: \"1e62999c-0ddd-410b-817a-3b744196df42\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:50.087209 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:50.087188 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1e62999c-0ddd-410b-817a-3b744196df42-data\") pod \"postgres-69fb4594cf-sdmv5\" (UID: \"1e62999c-0ddd-410b-817a-3b744196df42\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:50.094420 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:50.094391 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbdt\" (UniqueName: \"kubernetes.io/projected/1e62999c-0ddd-410b-817a-3b744196df42-kube-api-access-5lbdt\") pod \"postgres-69fb4594cf-sdmv5\" (UID: \"1e62999c-0ddd-410b-817a-3b744196df42\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:50.195561 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:50.195478 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:50.328758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:50.328725 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5"] Jul 23 23:20:50.332252 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:20:50.332226 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e62999c_0ddd_410b_817a_3b744196df42.slice/crio-99622278f2c0645a592bb6bd645b2b37bb2a12e14580847886d14ecc1b4d94f9 WatchSource:0}: Error finding container 99622278f2c0645a592bb6bd645b2b37bb2a12e14580847886d14ecc1b4d94f9: Status 404 returned error can't find the container with id 99622278f2c0645a592bb6bd645b2b37bb2a12e14580847886d14ecc1b4d94f9 Jul 23 23:20:50.367158 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:50.367132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" event={"ID":"1e62999c-0ddd-410b-817a-3b744196df42","Type":"ContainerStarted","Data":"99622278f2c0645a592bb6bd645b2b37bb2a12e14580847886d14ecc1b4d94f9"} Jul 23 23:20:51.170790 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:51.170751 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281d1043-3d93-4f4b-bc3f-c474fdb8bd7f" path="/var/lib/kubelet/pods/281d1043-3d93-4f4b-bc3f-c474fdb8bd7f/volumes" Jul 23 23:20:55.403671 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:55.403590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" event={"ID":"1e62999c-0ddd-410b-817a-3b744196df42","Type":"ContainerStarted","Data":"866cb20152334e32377ad7bf610fafb4343bb1c02576a72e18ee578a489c205e"} Jul 23 23:20:55.404091 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:55.403684 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:20:55.419546 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:20:55.419500 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" podStartSLOduration=1.6756718670000001 podStartE2EDuration="6.41948825s" totalImagesPullingTime="4.743816383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:20:55.417354564 +0000 UTC m=+478.945631731" watchObservedRunningTime="2026-07-23 23:20:55.41948825 +0000 UTC m=+478.947765427" Jul 23 23:21:01.436250 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:01.436218 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-sdmv5" Jul 23 23:21:01.755513 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:01.755428 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:02.371007 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:02.370968 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:08.915292 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:08.915251 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:08.919368 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:08.919342 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b6855cfc5-mpzjq"] Jul 23 23:21:08.949709 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:08.949676 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6855cfc5-mpzjq"] Jul 23 23:21:08.949852 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:08.949766 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:08.952547 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:08.952520 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 23:21:08.952547 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:08.952534 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wkkkf\"" Jul 23 23:21:08.952760 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:08.952553 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 23:21:09.016680 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.016634 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:09.044201 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.044178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5grr\" (UniqueName: \"kubernetes.io/projected/2bf56738-4114-4944-aba2-b5b9a266094e-kube-api-access-v5grr\") pod \"maas-controller-6b6855cfc5-mpzjq\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.044298 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.044222 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert\") pod \"maas-controller-6b6855cfc5-mpzjq\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.070102 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.070076 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:09.105930 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.105904 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79fccc4f95-j96lz"] Jul 23 23:21:09.109449 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.109429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.117603 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.117581 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-j96lz"] Jul 23 23:21:09.144818 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.144789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5grr\" (UniqueName: \"kubernetes.io/projected/2bf56738-4114-4944-aba2-b5b9a266094e-kube-api-access-v5grr\") pod \"maas-controller-6b6855cfc5-mpzjq\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.144972 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.144825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert\") pod \"maas-controller-6b6855cfc5-mpzjq\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.144972 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:21:09.144947 2568 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 23 23:21:09.145104 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:21:09.144995 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert podName:2bf56738-4114-4944-aba2-b5b9a266094e nodeName:}" failed. No retries permitted until 2026-07-23 23:21:09.644979524 +0000 UTC m=+493.173256669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert") pod "maas-controller-6b6855cfc5-mpzjq" (UID: "2bf56738-4114-4944-aba2-b5b9a266094e") : secret "maas-controller-webhook-cert" not found Jul 23 23:21:09.154538 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.154511 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5grr\" (UniqueName: \"kubernetes.io/projected/2bf56738-4114-4944-aba2-b5b9a266094e-kube-api-access-v5grr\") pod \"maas-controller-6b6855cfc5-mpzjq\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.245632 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.245558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5398408-b57f-4d05-b359-8a4da2b09ce5-webhook-cert\") pod \"maas-controller-79fccc4f95-j96lz\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.245632 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.245603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254hb\" (UniqueName: \"kubernetes.io/projected/e5398408-b57f-4d05-b359-8a4da2b09ce5-kube-api-access-254hb\") pod \"maas-controller-79fccc4f95-j96lz\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.335267 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.335239 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:09.346348 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.346321 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5398408-b57f-4d05-b359-8a4da2b09ce5-webhook-cert\") pod \"maas-controller-79fccc4f95-j96lz\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.346459 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.346350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254hb\" (UniqueName: \"kubernetes.io/projected/e5398408-b57f-4d05-b359-8a4da2b09ce5-kube-api-access-254hb\") pod \"maas-controller-79fccc4f95-j96lz\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.348709 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.348683 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5398408-b57f-4d05-b359-8a4da2b09ce5-webhook-cert\") pod \"maas-controller-79fccc4f95-j96lz\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.354027 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.354009 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-254hb\" (UniqueName: \"kubernetes.io/projected/e5398408-b57f-4d05-b359-8a4da2b09ce5-kube-api-access-254hb\") pod \"maas-controller-79fccc4f95-j96lz\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.419999 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.419976 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:09.548495 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.548436 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-j96lz"] Jul 23 23:21:09.550773 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:21:09.550751 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5398408_b57f_4d05_b359_8a4da2b09ce5.slice/crio-443d4814bddd2845fd47d81c765eefec54c6421751de1dee09b1d3a09afc6eb4 WatchSource:0}: Error finding container 443d4814bddd2845fd47d81c765eefec54c6421751de1dee09b1d3a09afc6eb4: Status 404 returned error can't find the container with id 443d4814bddd2845fd47d81c765eefec54c6421751de1dee09b1d3a09afc6eb4 Jul 23 23:21:09.649247 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.649218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert\") pod \"maas-controller-6b6855cfc5-mpzjq\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.651459 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.651434 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert\") pod \"maas-controller-6b6855cfc5-mpzjq\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.861168 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.861139 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:09.989725 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:09.989701 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6855cfc5-mpzjq"] Jul 23 23:21:09.991713 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:21:09.991684 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf56738_4114_4944_aba2_b5b9a266094e.slice/crio-032dbd56f0ea23e6d474ca8e54b6ff2c827be676a5329187a729de2ef4717cee WatchSource:0}: Error finding container 032dbd56f0ea23e6d474ca8e54b6ff2c827be676a5329187a729de2ef4717cee: Status 404 returned error can't find the container with id 032dbd56f0ea23e6d474ca8e54b6ff2c827be676a5329187a729de2ef4717cee Jul 23 23:21:10.474113 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:10.474074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-j96lz" event={"ID":"e5398408-b57f-4d05-b359-8a4da2b09ce5","Type":"ContainerStarted","Data":"443d4814bddd2845fd47d81c765eefec54c6421751de1dee09b1d3a09afc6eb4"} Jul 23 23:21:10.475479 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:10.475439 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" event={"ID":"2bf56738-4114-4944-aba2-b5b9a266094e","Type":"ContainerStarted","Data":"032dbd56f0ea23e6d474ca8e54b6ff2c827be676a5329187a729de2ef4717cee"} Jul 23 23:21:12.297580 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.297543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:12.393971 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.393888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:12.493378 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.493336 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-j96lz" event={"ID":"e5398408-b57f-4d05-b359-8a4da2b09ce5","Type":"ContainerStarted","Data":"6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637"} Jul 23 23:21:12.493683 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.493652 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:12.494978 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.494953 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" event={"ID":"2bf56738-4114-4944-aba2-b5b9a266094e","Type":"ContainerStarted","Data":"87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b"} Jul 23 23:21:12.495075 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.494995 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:12.517587 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.517548 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79fccc4f95-j96lz" podStartSLOduration=0.937333202 podStartE2EDuration="3.517535748s" totalImagesPullingTime="2.580202546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:21:12.51624108 +0000 UTC m=+496.044518260" watchObservedRunningTime="2026-07-23 23:21:12.517535748 +0000 UTC m=+496.045812915" Jul 23 23:21:12.532182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:12.532141 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" podStartSLOduration=2.386647556 podStartE2EDuration="4.532128433s" totalImagesPullingTime="2.145480877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:21:12.53081554 +0000 UTC m=+496.059092718" watchObservedRunningTime="2026-07-23 23:21:12.532128433 +0000 UTC m=+496.060405599" Jul 23 23:21:23.501846 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.501813 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:21:23.502242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.501902 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:23.568624 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.568590 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6855cfc5-mpzjq"] Jul 23 23:21:23.568848 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.568797 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" podUID="2bf56738-4114-4944-aba2-b5b9a266094e" containerName="manager" containerID="cri-o://87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b" gracePeriod=30 Jul 23 23:21:23.820218 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.820192 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:23.978594 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.978559 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert\") pod \"2bf56738-4114-4944-aba2-b5b9a266094e\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " Jul 23 23:21:23.978755 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.978702 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5grr\" (UniqueName: \"kubernetes.io/projected/2bf56738-4114-4944-aba2-b5b9a266094e-kube-api-access-v5grr\") pod \"2bf56738-4114-4944-aba2-b5b9a266094e\" (UID: \"2bf56738-4114-4944-aba2-b5b9a266094e\") " Jul 23 23:21:23.980937 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.980906 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2bf56738-4114-4944-aba2-b5b9a266094e" (UID: "2bf56738-4114-4944-aba2-b5b9a266094e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:21:23.981123 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:23.981102 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf56738-4114-4944-aba2-b5b9a266094e-kube-api-access-v5grr" (OuterVolumeSpecName: "kube-api-access-v5grr") pod "2bf56738-4114-4944-aba2-b5b9a266094e" (UID: "2bf56738-4114-4944-aba2-b5b9a266094e"). InnerVolumeSpecName "kube-api-access-v5grr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:21:24.079336 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.079312 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5grr\" (UniqueName: \"kubernetes.io/projected/2bf56738-4114-4944-aba2-b5b9a266094e-kube-api-access-v5grr\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:21:24.079336 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.079337 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bf56738-4114-4944-aba2-b5b9a266094e-webhook-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:21:24.550702 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.550615 2568 generic.go:356] "Generic (PLEG): container finished" podID="2bf56738-4114-4944-aba2-b5b9a266094e" containerID="87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b" exitCode=0 Jul 23 23:21:24.550702 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.550654 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" event={"ID":"2bf56738-4114-4944-aba2-b5b9a266094e","Type":"ContainerDied","Data":"87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b"} Jul 23 23:21:24.550702 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.550659 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" Jul 23 23:21:24.550702 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.550695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6855cfc5-mpzjq" event={"ID":"2bf56738-4114-4944-aba2-b5b9a266094e","Type":"ContainerDied","Data":"032dbd56f0ea23e6d474ca8e54b6ff2c827be676a5329187a729de2ef4717cee"} Jul 23 23:21:24.551338 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.550715 2568 scope.go:117] "RemoveContainer" containerID="87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b" Jul 23 23:21:24.559878 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.559862 2568 scope.go:117] "RemoveContainer" containerID="87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b" Jul 23 23:21:24.560116 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:21:24.560098 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b\": container with ID starting with 87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b not found: ID does not exist" containerID="87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b" Jul 23 23:21:24.560162 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.560126 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b"} err="failed to get container status \"87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b\": rpc error: code = NotFound desc = could not find container \"87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b\": container with ID starting with 87e828cd5aa36fbd516131caee657a6f49605a583844e13716ec647ae9fac11b not found: ID does not exist" Jul 23 23:21:24.570251 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.570228 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6855cfc5-mpzjq"] Jul 23 23:21:24.575981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:24.575962 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b6855cfc5-mpzjq"] Jul 23 23:21:25.166022 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:25.165991 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf56738-4114-4944-aba2-b5b9a266094e" path="/var/lib/kubelet/pods/2bf56738-4114-4944-aba2-b5b9a266094e/volumes" Jul 23 23:21:43.834905 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:43.834865 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:43.935137 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:43.935099 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:44.489152 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:44.489117 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:44.738916 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:44.738868 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:44.894770 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:44.894738 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:45.003922 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.003889 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:45.027296 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.027264 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-s5wdx"] Jul 23 23:21:45.027752 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.027735 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf56738-4114-4944-aba2-b5b9a266094e" containerName="manager" Jul 23 23:21:45.027846 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.027754 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf56738-4114-4944-aba2-b5b9a266094e" containerName="manager" Jul 23 23:21:45.027901 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.027873 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf56738-4114-4944-aba2-b5b9a266094e" containerName="manager" Jul 23 23:21:45.032178 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.032158 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.041451 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.041405 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 23 23:21:45.041705 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.041687 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7mkbl\"" Jul 23 23:21:45.043529 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.043507 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-s5wdx"] Jul 23 23:21:45.140399 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.140368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnqcd\" (UniqueName: \"kubernetes.io/projected/27c8392a-402b-4436-950f-ab76923eb373-kube-api-access-cnqcd\") pod \"payload-processing-5bf47cd988-s5wdx\" (UID: \"27c8392a-402b-4436-950f-ab76923eb373\") " pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.140628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.140598 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c8392a-402b-4436-950f-ab76923eb373-plugins-config-volume\") pod \"payload-processing-5bf47cd988-s5wdx\" (UID: \"27c8392a-402b-4436-950f-ab76923eb373\") " pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.241957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.241877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c8392a-402b-4436-950f-ab76923eb373-plugins-config-volume\") pod \"payload-processing-5bf47cd988-s5wdx\" (UID: \"27c8392a-402b-4436-950f-ab76923eb373\") " pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.241957 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.241930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnqcd\" (UniqueName: \"kubernetes.io/projected/27c8392a-402b-4436-950f-ab76923eb373-kube-api-access-cnqcd\") pod \"payload-processing-5bf47cd988-s5wdx\" (UID: \"27c8392a-402b-4436-950f-ab76923eb373\") " pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.242625 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.242601 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c8392a-402b-4436-950f-ab76923eb373-plugins-config-volume\") pod \"payload-processing-5bf47cd988-s5wdx\" (UID: \"27c8392a-402b-4436-950f-ab76923eb373\") " pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.249095 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.249067 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnqcd\" (UniqueName: \"kubernetes.io/projected/27c8392a-402b-4436-950f-ab76923eb373-kube-api-access-cnqcd\") pod \"payload-processing-5bf47cd988-s5wdx\" (UID: \"27c8392a-402b-4436-950f-ab76923eb373\") " pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.343171 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.343137 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:45.478210 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.478183 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-s5wdx"] Jul 23 23:21:45.480172 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:21:45.480147 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c8392a_402b_4436_950f_ab76923eb373.slice/crio-153e1b0c7182e14056ceb871e9d012c259a6d2db899af3a7b6c19d48297ad348 WatchSource:0}: Error finding container 153e1b0c7182e14056ceb871e9d012c259a6d2db899af3a7b6c19d48297ad348: Status 404 returned error can't find the container with id 153e1b0c7182e14056ceb871e9d012c259a6d2db899af3a7b6c19d48297ad348 Jul 23 23:21:45.651834 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:45.651766 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" event={"ID":"27c8392a-402b-4436-950f-ab76923eb373","Type":"ContainerStarted","Data":"153e1b0c7182e14056ceb871e9d012c259a6d2db899af3a7b6c19d48297ad348"} Jul 23 23:21:47.664017 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:47.663982 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" event={"ID":"27c8392a-402b-4436-950f-ab76923eb373","Type":"ContainerStarted","Data":"aa27bab208c60f41b9e6c6289f02d127254c4de4de2619688c99701c95d525e9"} Jul 23 23:21:47.664383 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:47.664086 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:21:47.680500 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:47.680454 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" podStartSLOduration=0.582566066 podStartE2EDuration="2.680438288s" totalImagesPullingTime="2.097872222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:21:47.67859578 +0000 UTC m=+531.206872948" watchObservedRunningTime="2026-07-23 23:21:47.680438288 +0000 UTC m=+531.208715455" Jul 23 23:21:56.580173 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:56.580096 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:56.678843 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:56.678810 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:21:58.672255 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:21:58.672227 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-s5wdx" Jul 23 23:22:12.695171 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.695130 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54"] Jul 23 23:22:12.700028 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.700007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.702578 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.702554 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 23 23:22:12.702708 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.702614 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g6lpf\"" Jul 23 23:22:12.702708 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.702628 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 23:22:12.702708 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.702645 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 23:22:12.706380 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.706357 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54"] Jul 23 23:22:12.722666 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.722634 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:22:12.768893 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.768861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fkv\" (UniqueName: \"kubernetes.io/projected/348646c1-964d-4bb5-9a8d-a372e9edb1ba-kube-api-access-q7fkv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.769049 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.768905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.769049 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.768928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.769049 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.769022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.769193 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.769053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348646c1-964d-4bb5-9a8d-a372e9edb1ba-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.769193 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.769147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870474 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870632 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870632 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870632 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348646c1-964d-4bb5-9a8d-a372e9edb1ba-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870632 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870615 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870871 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870691 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fkv\" (UniqueName: \"kubernetes.io/projected/348646c1-964d-4bb5-9a8d-a372e9edb1ba-kube-api-access-q7fkv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870927 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.870988 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.870936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.871070 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.871003 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.872791 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.872773 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/348646c1-964d-4bb5-9a8d-a372e9edb1ba-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.873062 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.873046 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348646c1-964d-4bb5-9a8d-a372e9edb1ba-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:12.878002 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:12.877984 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fkv\" (UniqueName: \"kubernetes.io/projected/348646c1-964d-4bb5-9a8d-a372e9edb1ba-kube-api-access-q7fkv\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54\" (UID: \"348646c1-964d-4bb5-9a8d-a372e9edb1ba\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:13.012459 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:13.012337 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:13.147447 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:13.147420 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54"] Jul 23 23:22:13.150301 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:22:13.150273 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348646c1_964d_4bb5_9a8d_a372e9edb1ba.slice/crio-5d5416e87c05f485844cada170432a7dad5913a6d2ee59976d025d92fc971276 WatchSource:0}: Error finding container 5d5416e87c05f485844cada170432a7dad5913a6d2ee59976d025d92fc971276: Status 404 returned error can't find the container with id 5d5416e87c05f485844cada170432a7dad5913a6d2ee59976d025d92fc971276 Jul 23 23:22:13.786743 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:13.786703 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" event={"ID":"348646c1-964d-4bb5-9a8d-a372e9edb1ba","Type":"ContainerStarted","Data":"5d5416e87c05f485844cada170432a7dad5913a6d2ee59976d025d92fc971276"} Jul 23 23:22:14.793649 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:14.793559 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" event={"ID":"348646c1-964d-4bb5-9a8d-a372e9edb1ba","Type":"ContainerStarted","Data":"18ffb9d8a54965c6fb32b7c51bf9b442b456c4eefb5ba600c785531cea56b789"} Jul 23 23:22:14.810644 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:14.810595 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" podStartSLOduration=1.618056905 podStartE2EDuration="2.810580895s" totalImagesPullingTime="1.19252399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:22:14.809151447 +0000 UTC m=+558.337428613" watchObservedRunningTime="2026-07-23 23:22:14.810580895 +0000 UTC m=+558.338858070" Jul 23 23:22:23.012652 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:23.012612 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:23.013148 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:23.012704 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:23.025729 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:23.025709 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:23.846464 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:23.846438 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54" Jul 23 23:22:24.034021 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:24.033992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:22:26.045429 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:26.045369 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:22:32.434203 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:32.434160 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:22:33.478528 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:33.478493 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:22:48.704392 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.704356 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj"] Jul 23 23:22:48.711930 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.711907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.713718 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.713697 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 23:22:48.717572 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.717549 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj"] Jul 23 23:22:48.735909 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.735872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:22:48.874613 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.874583 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20022134-2259-4f38-a852-fab522a79ffa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.874613 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.874619 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.874816 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.874676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.874816 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.874694 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.874816 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.874727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xggs4\" (UniqueName: \"kubernetes.io/projected/20022134-2259-4f38-a852-fab522a79ffa-kube-api-access-xggs4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.874816 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.874767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975360 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975398 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975677 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975455 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xggs4\" (UniqueName: \"kubernetes.io/projected/20022134-2259-4f38-a852-fab522a79ffa-kube-api-access-xggs4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975677 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975677 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975519 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20022134-2259-4f38-a852-fab522a79ffa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975677 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975961 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975884 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.975961 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.975946 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.976075 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.976001 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.977632 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.977604 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20022134-2259-4f38-a852-fab522a79ffa-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.977942 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.977923 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20022134-2259-4f38-a852-fab522a79ffa-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:48.983117 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:48.983095 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xggs4\" (UniqueName: \"kubernetes.io/projected/20022134-2259-4f38-a852-fab522a79ffa-kube-api-access-xggs4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj\" (UID: \"20022134-2259-4f38-a852-fab522a79ffa\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:49.024036 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:49.024011 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:49.159087 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:49.159060 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj"] Jul 23 23:22:49.162231 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:22:49.162208 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20022134_2259_4f38_a852_fab522a79ffa.slice/crio-3c39d7e446def9110bf7f94f61a609f53c1d73751ad53eabee8b3afc00d6fbaf WatchSource:0}: Error finding container 3c39d7e446def9110bf7f94f61a609f53c1d73751ad53eabee8b3afc00d6fbaf: Status 404 returned error can't find the container with id 3c39d7e446def9110bf7f94f61a609f53c1d73751ad53eabee8b3afc00d6fbaf Jul 23 23:22:49.954274 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:49.954235 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" event={"ID":"20022134-2259-4f38-a852-fab522a79ffa","Type":"ContainerStarted","Data":"cec2326d427659e43d7e7398cfae1da0edceccf42250c71a36e68c588cda34a3"} Jul 23 23:22:49.954679 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:49.954320 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" event={"ID":"20022134-2259-4f38-a852-fab522a79ffa","Type":"ContainerStarted","Data":"3c39d7e446def9110bf7f94f61a609f53c1d73751ad53eabee8b3afc00d6fbaf"} Jul 23 23:22:49.970683 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:49.970638 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" podStartSLOduration=1.779740619 podStartE2EDuration="1.970613903s" totalImagesPullingTime="190.873284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:22:49.969149187 +0000 UTC m=+593.497426356" watchObservedRunningTime="2026-07-23 23:22:49.970613903 +0000 UTC m=+593.498891070" Jul 23 23:22:56.936961 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:56.936929 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:22:56.941738 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:56.941713 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:22:56.951238 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:56.951218 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:22:56.956261 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:56.956244 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:22:59.024284 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:59.024251 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:59.026677 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:59.024308 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:22:59.036616 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:22:59.036597 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:23:00.012991 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.012963 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj" Jul 23 23:23:00.708629 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.708595 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m"] Jul 23 23:23:00.712424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.712389 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.714246 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.714227 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 23 23:23:00.722813 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.722791 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m"] Jul 23 23:23:00.742906 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.742869 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:23:00.770486 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.770454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.770630 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.770510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8g82\" (UniqueName: \"kubernetes.io/projected/529f04f6-49bd-41c3-8187-74d669a4e2da-kube-api-access-c8g82\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.770630 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.770595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.770750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.770627 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.770750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.770659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529f04f6-49bd-41c3-8187-74d669a4e2da-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.770750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.770696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.871899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.871867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872054 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.871906 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g82\" (UniqueName: \"kubernetes.io/projected/529f04f6-49bd-41c3-8187-74d669a4e2da-kube-api-access-c8g82\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872054 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.871938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872162 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.872070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872162 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.872117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529f04f6-49bd-41c3-8187-74d669a4e2da-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872162 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.872158 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872363 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.872342 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872465 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.872448 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.872525 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.872508 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.874082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.874064 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/529f04f6-49bd-41c3-8187-74d669a4e2da-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.874595 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.874574 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/529f04f6-49bd-41c3-8187-74d669a4e2da-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:00.879048 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:00.879024 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8g82\" (UniqueName: \"kubernetes.io/projected/529f04f6-49bd-41c3-8187-74d669a4e2da-kube-api-access-c8g82\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r527m\" (UID: \"529f04f6-49bd-41c3-8187-74d669a4e2da\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:01.022829 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:01.022748 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:01.157382 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:01.157359 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m"] Jul 23 23:23:01.159587 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:23:01.159554 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529f04f6_49bd_41c3_8187_74d669a4e2da.slice/crio-a1182683291a9740d74de1f092b1c26b9894999116a8e0f1b23d7019ae1e8131 WatchSource:0}: Error finding container a1182683291a9740d74de1f092b1c26b9894999116a8e0f1b23d7019ae1e8131: Status 404 returned error can't find the container with id a1182683291a9740d74de1f092b1c26b9894999116a8e0f1b23d7019ae1e8131 Jul 23 23:23:01.161090 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:01.161071 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 23:23:02.019009 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:02.018969 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" event={"ID":"529f04f6-49bd-41c3-8187-74d669a4e2da","Type":"ContainerStarted","Data":"a53f9e3b7435e3cfbe29d63c3378fda5a54e2e57ba93e880e26c8e11bb09be76"} Jul 23 23:23:02.019437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:02.019030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" event={"ID":"529f04f6-49bd-41c3-8187-74d669a4e2da","Type":"ContainerStarted","Data":"a1182683291a9740d74de1f092b1c26b9894999116a8e0f1b23d7019ae1e8131"} Jul 23 23:23:02.036684 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:02.036636 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" podStartSLOduration=1.8016840950000002 podStartE2EDuration="2.036620451s" totalImagesPullingTime="234.936356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:23:02.035322353 +0000 UTC m=+605.563599521" watchObservedRunningTime="2026-07-23 23:23:02.036620451 +0000 UTC m=+605.564897669" Jul 23 23:23:07.523232 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:07.523183 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:23:11.023580 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:11.023544 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:11.023964 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:11.023620 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:11.036768 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:11.036743 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:11.076426 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:11.076380 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r527m" Jul 23 23:23:28.850866 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:28.850787 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:23:39.513986 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:23:39.513948 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:24:03.518123 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:03.518086 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:24:03.623369 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:03.623324 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:24:04.696972 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:04.696925 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:24:04.796396 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:04.796358 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:24:50.915182 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:50.915083 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-j96lz"] Jul 23 23:24:50.915763 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:50.915342 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79fccc4f95-j96lz" podUID="e5398408-b57f-4d05-b359-8a4da2b09ce5" containerName="manager" containerID="cri-o://6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637" gracePeriod=30 Jul 23 23:24:51.182576 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.182551 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:24:51.250181 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.250149 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-254hb\" (UniqueName: \"kubernetes.io/projected/e5398408-b57f-4d05-b359-8a4da2b09ce5-kube-api-access-254hb\") pod \"e5398408-b57f-4d05-b359-8a4da2b09ce5\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " Jul 23 23:24:51.250344 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.250194 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5398408-b57f-4d05-b359-8a4da2b09ce5-webhook-cert\") pod \"e5398408-b57f-4d05-b359-8a4da2b09ce5\" (UID: \"e5398408-b57f-4d05-b359-8a4da2b09ce5\") " Jul 23 23:24:51.252319 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.252291 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5398408-b57f-4d05-b359-8a4da2b09ce5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e5398408-b57f-4d05-b359-8a4da2b09ce5" (UID: "e5398408-b57f-4d05-b359-8a4da2b09ce5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:24:51.252449 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.252340 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5398408-b57f-4d05-b359-8a4da2b09ce5-kube-api-access-254hb" (OuterVolumeSpecName: "kube-api-access-254hb") pod "e5398408-b57f-4d05-b359-8a4da2b09ce5" (UID: "e5398408-b57f-4d05-b359-8a4da2b09ce5"). InnerVolumeSpecName "kube-api-access-254hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:24:51.350991 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.350951 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254hb\" (UniqueName: \"kubernetes.io/projected/e5398408-b57f-4d05-b359-8a4da2b09ce5-kube-api-access-254hb\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:24:51.350991 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.350983 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5398408-b57f-4d05-b359-8a4da2b09ce5-webhook-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:24:51.558563 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.558470 2568 generic.go:356] "Generic (PLEG): container finished" podID="e5398408-b57f-4d05-b359-8a4da2b09ce5" containerID="6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637" exitCode=0 Jul 23 23:24:51.558563 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.558516 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-j96lz" Jul 23 23:24:51.558563 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.558554 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-j96lz" event={"ID":"e5398408-b57f-4d05-b359-8a4da2b09ce5","Type":"ContainerDied","Data":"6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637"} Jul 23 23:24:51.558777 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.558610 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-j96lz" event={"ID":"e5398408-b57f-4d05-b359-8a4da2b09ce5","Type":"ContainerDied","Data":"443d4814bddd2845fd47d81c765eefec54c6421751de1dee09b1d3a09afc6eb4"} Jul 23 23:24:51.558777 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.558634 2568 scope.go:117] "RemoveContainer" containerID="6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637" Jul 23 23:24:51.568449 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.568429 2568 scope.go:117] "RemoveContainer" containerID="6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637" Jul 23 23:24:51.568698 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:24:51.568679 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637\": container with ID starting with 6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637 not found: ID does not exist" containerID="6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637" Jul 23 23:24:51.568748 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.568707 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637"} err="failed to get container status \"6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637\": rpc error: code = NotFound desc = could not find container \"6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637\": container with ID starting with 6b5b8395e6f71782867b0a972a89ab3512db19dffa7ff6d60085f20fc4922637 not found: ID does not exist" Jul 23 23:24:51.580012 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.579988 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-j96lz"] Jul 23 23:24:51.581703 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:51.581681 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-j96lz"] Jul 23 23:24:53.168595 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:24:53.167051 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5398408-b57f-4d05-b359-8a4da2b09ce5" path="/var/lib/kubelet/pods/e5398408-b57f-4d05-b359-8a4da2b09ce5/volumes" Jul 23 23:25:51.678046 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:51.678016 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:25:51.765252 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:51.765218 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:25:52.428225 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.428192 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79fccc4f95-x4nk7"] Jul 23 23:25:52.428628 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.428615 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5398408-b57f-4d05-b359-8a4da2b09ce5" containerName="manager" Jul 23 23:25:52.428688 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.428629 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5398408-b57f-4d05-b359-8a4da2b09ce5" containerName="manager" Jul 23 23:25:52.428725 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.428696 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5398408-b57f-4d05-b359-8a4da2b09ce5" containerName="manager" Jul 23 23:25:52.431824 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.431804 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.433776 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.433756 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 23:25:52.434262 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.434242 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wkkkf\"" Jul 23 23:25:52.434369 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.434292 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 23:25:52.442047 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.442027 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-x4nk7"] Jul 23 23:25:52.586799 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.586754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-webhook-cert\") pod \"maas-controller-79fccc4f95-x4nk7\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.586993 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.586887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwd6\" (UniqueName: \"kubernetes.io/projected/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-kube-api-access-8cwd6\") pod \"maas-controller-79fccc4f95-x4nk7\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.687735 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.687655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-webhook-cert\") pod \"maas-controller-79fccc4f95-x4nk7\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.687735 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.687725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwd6\" (UniqueName: \"kubernetes.io/projected/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-kube-api-access-8cwd6\") pod \"maas-controller-79fccc4f95-x4nk7\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.690135 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.690109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-webhook-cert\") pod \"maas-controller-79fccc4f95-x4nk7\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.695402 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.695377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwd6\" (UniqueName: \"kubernetes.io/projected/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-kube-api-access-8cwd6\") pod \"maas-controller-79fccc4f95-x4nk7\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.743273 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.743240 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:52.876143 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:52.876085 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-x4nk7"] Jul 23 23:25:52.878617 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:25:52.878589 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fddc1b4_6da5_450d_9b3f_94fbb72eb2c6.slice/crio-05b097946bda4780b269b22f184361c0cf22e5ed1aa020a24749da34ad3c3006 WatchSource:0}: Error finding container 05b097946bda4780b269b22f184361c0cf22e5ed1aa020a24749da34ad3c3006: Status 404 returned error can't find the container with id 05b097946bda4780b269b22f184361c0cf22e5ed1aa020a24749da34ad3c3006 Jul 23 23:25:53.855859 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:53.855828 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" event={"ID":"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6","Type":"ContainerStarted","Data":"1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc"} Jul 23 23:25:53.856274 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:53.855877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" event={"ID":"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6","Type":"ContainerStarted","Data":"05b097946bda4780b269b22f184361c0cf22e5ed1aa020a24749da34ad3c3006"} Jul 23 23:25:53.856274 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:53.855940 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:25:53.909902 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:25:53.909854 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" podStartSLOduration=1.414325918 podStartE2EDuration="1.909840756s" totalImagesPullingTime="495.514838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:25:53.908170724 +0000 UTC m=+777.436447892" watchObservedRunningTime="2026-07-23 23:25:53.909840756 +0000 UTC m=+777.438117922" Jul 23 23:26:04.862546 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:04.862509 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:26:09.429095 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.429060 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:09.520578 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.520546 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-x4nk7"] Jul 23 23:26:09.520799 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.520774 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" podUID="0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" containerName="manager" containerID="cri-o://1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc" gracePeriod=30 Jul 23 23:26:09.778858 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.778836 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:26:09.935394 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.935355 2568 generic.go:356] "Generic (PLEG): container finished" podID="0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" containerID="1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc" exitCode=0 Jul 23 23:26:09.935565 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.935399 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" Jul 23 23:26:09.935565 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.935436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" event={"ID":"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6","Type":"ContainerDied","Data":"1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc"} Jul 23 23:26:09.935565 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.935495 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-x4nk7" event={"ID":"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6","Type":"ContainerDied","Data":"05b097946bda4780b269b22f184361c0cf22e5ed1aa020a24749da34ad3c3006"} Jul 23 23:26:09.935565 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.935528 2568 scope.go:117] "RemoveContainer" containerID="1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc" Jul 23 23:26:09.943953 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.943932 2568 scope.go:117] "RemoveContainer" containerID="1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc" Jul 23 23:26:09.944198 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:26:09.944174 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc\": container with ID starting with 1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc not found: ID does not exist" containerID="1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc" Jul 23 23:26:09.944266 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.944202 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc"} err="failed to get container status \"1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc\": rpc error: code = NotFound desc = could not find container \"1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc\": container with ID starting with 1b8d584b42c54465c04be92962e667da9822e6a3a7ba497a2180236ddc0249cc not found: ID does not exist" Jul 23 23:26:09.945459 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.945443 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cwd6\" (UniqueName: \"kubernetes.io/projected/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-kube-api-access-8cwd6\") pod \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " Jul 23 23:26:09.945546 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.945534 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-webhook-cert\") pod \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\" (UID: \"0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6\") " Jul 23 23:26:09.947244 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.947219 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-kube-api-access-8cwd6" (OuterVolumeSpecName: "kube-api-access-8cwd6") pod "0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" (UID: "0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6"). InnerVolumeSpecName "kube-api-access-8cwd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:26:09.947619 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:09.947601 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" (UID: "0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 23:26:10.046281 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:10.046245 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cwd6\" (UniqueName: \"kubernetes.io/projected/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-kube-api-access-8cwd6\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:26:10.046281 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:10.046272 2568 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6-webhook-cert\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:26:10.256772 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:10.256739 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-x4nk7"] Jul 23 23:26:10.260076 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:10.260049 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-x4nk7"] Jul 23 23:26:11.042621 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.042591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:11.167601 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.167568 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" path="/var/lib/kubelet/pods/0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6/volumes" Jul 23 23:26:11.167884 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.167867 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79fccc4f95-h6rn8"] Jul 23 23:26:11.168277 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.168262 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" containerName="manager" Jul 23 23:26:11.168330 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.168278 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" containerName="manager" Jul 23 23:26:11.168369 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.168360 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fddc1b4-6da5-450d-9b3f-94fbb72eb2c6" containerName="manager" Jul 23 23:26:11.171510 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.171495 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.175528 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.175330 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 23 23:26:11.175528 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.175330 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wkkkf\"" Jul 23 23:26:11.175528 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.175338 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 23 23:26:11.177668 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.177649 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-h6rn8"] Jul 23 23:26:11.257057 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.257027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69a21b07-4fe7-48e7-a626-7d492ea5066a-webhook-cert\") pod \"maas-controller-79fccc4f95-h6rn8\" (UID: \"69a21b07-4fe7-48e7-a626-7d492ea5066a\") " pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.257216 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.257077 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9kx\" (UniqueName: \"kubernetes.io/projected/69a21b07-4fe7-48e7-a626-7d492ea5066a-kube-api-access-4h9kx\") pod \"maas-controller-79fccc4f95-h6rn8\" (UID: \"69a21b07-4fe7-48e7-a626-7d492ea5066a\") " pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.358147 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.358117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69a21b07-4fe7-48e7-a626-7d492ea5066a-webhook-cert\") pod \"maas-controller-79fccc4f95-h6rn8\" (UID: \"69a21b07-4fe7-48e7-a626-7d492ea5066a\") " pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.358342 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.358184 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9kx\" (UniqueName: \"kubernetes.io/projected/69a21b07-4fe7-48e7-a626-7d492ea5066a-kube-api-access-4h9kx\") pod \"maas-controller-79fccc4f95-h6rn8\" (UID: \"69a21b07-4fe7-48e7-a626-7d492ea5066a\") " pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.360621 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.360593 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69a21b07-4fe7-48e7-a626-7d492ea5066a-webhook-cert\") pod \"maas-controller-79fccc4f95-h6rn8\" (UID: \"69a21b07-4fe7-48e7-a626-7d492ea5066a\") " pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.366668 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.366647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9kx\" (UniqueName: \"kubernetes.io/projected/69a21b07-4fe7-48e7-a626-7d492ea5066a-kube-api-access-4h9kx\") pod \"maas-controller-79fccc4f95-h6rn8\" (UID: \"69a21b07-4fe7-48e7-a626-7d492ea5066a\") " pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.484543 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.484513 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:11.617781 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.617755 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79fccc4f95-h6rn8"] Jul 23 23:26:11.620506 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:26:11.620480 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a21b07_4fe7_48e7_a626_7d492ea5066a.slice/crio-ae4424cc733ccdfe0a982a9380da547ff47ab76383a67faa1a7bc552a4ccfff1 WatchSource:0}: Error finding container ae4424cc733ccdfe0a982a9380da547ff47ab76383a67faa1a7bc552a4ccfff1: Status 404 returned error can't find the container with id ae4424cc733ccdfe0a982a9380da547ff47ab76383a67faa1a7bc552a4ccfff1 Jul 23 23:26:11.948021 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:11.947938 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-h6rn8" event={"ID":"69a21b07-4fe7-48e7-a626-7d492ea5066a","Type":"ContainerStarted","Data":"ae4424cc733ccdfe0a982a9380da547ff47ab76383a67faa1a7bc552a4ccfff1"} Jul 23 23:26:12.954002 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:12.953963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79fccc4f95-h6rn8" event={"ID":"69a21b07-4fe7-48e7-a626-7d492ea5066a","Type":"ContainerStarted","Data":"4d28c4c204d7b45f59d229c0210ff11109098cc42d96fa0d6f4fa357403c4c92"} Jul 23 23:26:12.954365 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:12.954043 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:12.970347 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:12.970303 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79fccc4f95-h6rn8" podStartSLOduration=1.589333957 podStartE2EDuration="1.970289091s" totalImagesPullingTime="380.955134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:26:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:26:12.969155161 +0000 UTC m=+796.497432329" watchObservedRunningTime="2026-07-23 23:26:12.970289091 +0000 UTC m=+796.498566258" Jul 23 23:26:23.961543 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:23.961454 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79fccc4f95-h6rn8" Jul 23 23:26:47.292792 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:47.292750 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:47.423596 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:47.423561 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:47.541082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:47.541049 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:53.378376 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:53.378339 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:53.440424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:53.440369 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:53.529498 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:53.529444 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:53.629403 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:53.629312 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:53.729077 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:53.729039 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:59.460796 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:59.460763 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:26:59.516631 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:26:59.516578 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:27:04.568115 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:27:04.568084 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:27:04.668305 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:27:04.668278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:27:56.975195 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:27:56.975118 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:27:56.979478 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:27:56.979458 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:27:56.983176 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:27:56.983159 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:27:56.987196 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:27:56.987177 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:28:11.411139 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:28:11.411102 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:29:15.453736 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:29:15.453699 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:29:40.563531 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:29:40.563490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:29:40.664090 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:29:40.664037 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:30:55.319153 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:55.319050 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46"] Jul 23 23:30:55.319721 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:55.319247 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" podUID="b8a5885d-cde8-4e3e-8e01-b7950d239bbf" containerName="manager" containerID="cri-o://fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce" gracePeriod=10 Jul 23 23:30:57.769202 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:57.769176 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:30:57.881728 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:57.881701 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-extensions-socket-volume\") pod \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " Jul 23 23:30:57.881874 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:57.881784 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j8jh\" (UniqueName: \"kubernetes.io/projected/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-kube-api-access-2j8jh\") pod \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\" (UID: \"b8a5885d-cde8-4e3e-8e01-b7950d239bbf\") " Jul 23 23:30:57.882058 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:57.882034 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8a5885d-cde8-4e3e-8e01-b7950d239bbf" (UID: "b8a5885d-cde8-4e3e-8e01-b7950d239bbf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 23:30:57.883695 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:57.883669 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-kube-api-access-2j8jh" (OuterVolumeSpecName: "kube-api-access-2j8jh") pod "b8a5885d-cde8-4e3e-8e01-b7950d239bbf" (UID: "b8a5885d-cde8-4e3e-8e01-b7950d239bbf"). InnerVolumeSpecName "kube-api-access-2j8jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:30:57.982791 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:57.982764 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-extensions-socket-volume\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:30:57.982791 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:57.982788 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8jh\" (UniqueName: \"kubernetes.io/projected/b8a5885d-cde8-4e3e-8e01-b7950d239bbf-kube-api-access-2j8jh\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:30:58.390751 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.390721 2568 generic.go:356] "Generic (PLEG): container finished" podID="b8a5885d-cde8-4e3e-8e01-b7950d239bbf" containerID="fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce" exitCode=0 Jul 23 23:30:58.390908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.390764 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" Jul 23 23:30:58.390908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.390803 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" event={"ID":"b8a5885d-cde8-4e3e-8e01-b7950d239bbf","Type":"ContainerDied","Data":"fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce"} Jul 23 23:30:58.390908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.390856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46" event={"ID":"b8a5885d-cde8-4e3e-8e01-b7950d239bbf","Type":"ContainerDied","Data":"33defbddb9640a62de05798afbce52bd9a69ee855768a1948b0173fd05d57444"} Jul 23 23:30:58.390908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.390880 2568 scope.go:117] "RemoveContainer" containerID="fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce" Jul 23 23:30:58.400467 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.400450 2568 scope.go:117] "RemoveContainer" containerID="fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce" Jul 23 23:30:58.400718 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:30:58.400701 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce\": container with ID starting with fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce not found: ID does not exist" containerID="fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce" Jul 23 23:30:58.400758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.400727 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce"} err="failed to get container status \"fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce\": rpc error: code = NotFound desc = could not find container \"fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce\": container with ID starting with fc4900de272162b28b4d77a50eada5ce4f228fa57339322f6c485398d16ab8ce not found: ID does not exist" Jul 23 23:30:58.410136 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.410116 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46"] Jul 23 23:30:58.414039 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:58.414021 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vgr46"] Jul 23 23:30:59.166557 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:30:59.166526 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a5885d-cde8-4e3e-8e01-b7950d239bbf" path="/var/lib/kubelet/pods/b8a5885d-cde8-4e3e-8e01-b7950d239bbf/volumes" Jul 23 23:31:56.495929 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.495896 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588"] Jul 23 23:31:56.496370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.496283 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a5885d-cde8-4e3e-8e01-b7950d239bbf" containerName="manager" Jul 23 23:31:56.496370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.496294 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a5885d-cde8-4e3e-8e01-b7950d239bbf" containerName="manager" Jul 23 23:31:56.496370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.496359 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a5885d-cde8-4e3e-8e01-b7950d239bbf" containerName="manager" Jul 23 23:31:56.499457 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.499439 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.501429 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.501391 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m2lst\"" Jul 23 23:31:56.512027 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.512007 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588"] Jul 23 23:31:56.556758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.556735 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7cee4f6-a438-48d3-9d01-3fdb299ecbf1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cd588\" (UID: \"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.556879 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.556767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6wr\" (UniqueName: \"kubernetes.io/projected/f7cee4f6-a438-48d3-9d01-3fdb299ecbf1-kube-api-access-xf6wr\") pod \"kuadrant-operator-controller-manager-86878c769f-cd588\" (UID: \"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.657619 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.657596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7cee4f6-a438-48d3-9d01-3fdb299ecbf1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cd588\" (UID: \"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.657743 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.657627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6wr\" (UniqueName: \"kubernetes.io/projected/f7cee4f6-a438-48d3-9d01-3fdb299ecbf1-kube-api-access-xf6wr\") pod \"kuadrant-operator-controller-manager-86878c769f-cd588\" (UID: \"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.657977 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.657955 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7cee4f6-a438-48d3-9d01-3fdb299ecbf1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cd588\" (UID: \"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.664921 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.664903 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6wr\" (UniqueName: \"kubernetes.io/projected/f7cee4f6-a438-48d3-9d01-3fdb299ecbf1-kube-api-access-xf6wr\") pod \"kuadrant-operator-controller-manager-86878c769f-cd588\" (UID: \"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.809275 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.809200 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:56.951708 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:56.951678 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588"] Jul 23 23:31:56.955373 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:31:56.955341 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cee4f6_a438_48d3_9d01_3fdb299ecbf1.slice/crio-35a49f7f1ce42fb5001cd553a8ea13e517d138ca5a2a95429c334bd21368765a WatchSource:0}: Error finding container 35a49f7f1ce42fb5001cd553a8ea13e517d138ca5a2a95429c334bd21368765a: Status 404 returned error can't find the container with id 35a49f7f1ce42fb5001cd553a8ea13e517d138ca5a2a95429c334bd21368765a Jul 23 23:31:57.692209 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:57.692171 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" event={"ID":"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1","Type":"ContainerStarted","Data":"15203cd4c5d2fc14d89536855272945466e2ca2e4a55996e438cbd795d82c378"} Jul 23 23:31:57.692600 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:57.692236 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" event={"ID":"f7cee4f6-a438-48d3-9d01-3fdb299ecbf1","Type":"ContainerStarted","Data":"35a49f7f1ce42fb5001cd553a8ea13e517d138ca5a2a95429c334bd21368765a"} Jul 23 23:31:57.692600 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:57.692353 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:31:57.708600 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:31:57.708556 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" podStartSLOduration=1.708543888 podStartE2EDuration="1.708543888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:31:57.707795226 +0000 UTC m=+1141.236072394" watchObservedRunningTime="2026-07-23 23:31:57.708543888 +0000 UTC m=+1141.236821117" Jul 23 23:32:08.699512 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:32:08.699483 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cd588" Jul 23 23:32:57.008174 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:32:57.008138 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:32:57.008714 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:32:57.008444 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:32:57.015942 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:32:57.015921 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:32:57.016645 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:32:57.016627 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:32:59.687103 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:32:59.687059 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:34:31.360707 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:34:31.360669 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:34:34.923380 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:34:34.923348 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:35:01.473424 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:35:01.473378 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:35:02.694639 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:35:02.694601 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:37:57.042373 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:37:57.042343 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:37:57.044004 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:37:57.043978 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:37:57.050101 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:37:57.050077 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:37:57.051397 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:37:57.051376 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:38:03.577992 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:03.577958 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:24.230475 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:24.230385 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:27.405211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:27.405180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:27.506914 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:27.506883 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:27.872759 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:27.872726 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:33.705293 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:33.705255 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:33.806813 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:33.806779 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:33.943506 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:33.943472 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:34.254080 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:34.254046 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:34.515022 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:34.514940 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:36.471896 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:36.471862 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:36.552138 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:36.552106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:36.615970 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:36.615933 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:36.887107 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:36.887075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:37.054798 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:37.054768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:38:37.155937 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:38:37.155851 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:00.069842 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:00.069810 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:02.093241 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.093207 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2"] Jul 23 23:39:02.096662 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.096646 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" Jul 23 23:39:02.098370 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.098348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:39:02.102232 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.102208 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2"] Jul 23 23:39:02.235994 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.235961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86021cdd-6660-47bd-a5f2-1566b15e600b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2\" (UID: \"86021cdd-6660-47bd-a5f2-1566b15e600b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" Jul 23 23:39:02.337543 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.337511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86021cdd-6660-47bd-a5f2-1566b15e600b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2\" (UID: \"86021cdd-6660-47bd-a5f2-1566b15e600b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" Jul 23 23:39:02.338201 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.338181 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86021cdd-6660-47bd-a5f2-1566b15e600b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2\" (UID: \"86021cdd-6660-47bd-a5f2-1566b15e600b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" Jul 23 23:39:02.410699 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.410627 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" Jul 23 23:39:02.544623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.544599 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2"] Jul 23 23:39:02.547092 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:39:02.547068 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86021cdd_6660_47bd_a5f2_1566b15e600b.slice/crio-b620bc215313d322ded0a1fcd67f02d842551dc39e3cdd0f8a10a2f0c326e76c WatchSource:0}: Error finding container b620bc215313d322ded0a1fcd67f02d842551dc39e3cdd0f8a10a2f0c326e76c: Status 404 returned error can't find the container with id b620bc215313d322ded0a1fcd67f02d842551dc39e3cdd0f8a10a2f0c326e76c Jul 23 23:39:02.548955 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.548935 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 23:39:02.796788 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:02.796695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" event={"ID":"86021cdd-6660-47bd-a5f2-1566b15e600b","Type":"ContainerStarted","Data":"b620bc215313d322ded0a1fcd67f02d842551dc39e3cdd0f8a10a2f0c326e76c"} Jul 23 23:39:05.813557 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:05.813461 2568 generic.go:356] "Generic (PLEG): container finished" podID="86021cdd-6660-47bd-a5f2-1566b15e600b" containerID="59d4ca57769b2065ae6a4c7c2ceb22796c1bac62d97b91b4d3c2f5ed8232ffc7" exitCode=0 Jul 23 23:39:05.813966 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:05.813556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" event={"ID":"86021cdd-6660-47bd-a5f2-1566b15e600b","Type":"ContainerDied","Data":"59d4ca57769b2065ae6a4c7c2ceb22796c1bac62d97b91b4d3c2f5ed8232ffc7"} Jul 23 23:39:06.957479 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:06.957454 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" Jul 23 23:39:07.080580 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:07.080513 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86021cdd-6660-47bd-a5f2-1566b15e600b-maas-api-service-ca\") pod \"86021cdd-6660-47bd-a5f2-1566b15e600b\" (UID: \"86021cdd-6660-47bd-a5f2-1566b15e600b\") " Jul 23 23:39:07.080891 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:07.080862 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86021cdd-6660-47bd-a5f2-1566b15e600b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86021cdd-6660-47bd-a5f2-1566b15e600b" (UID: "86021cdd-6660-47bd-a5f2-1566b15e600b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:39:07.181666 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:07.181641 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86021cdd-6660-47bd-a5f2-1566b15e600b-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:39:07.825981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:07.825949 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" Jul 23 23:39:07.825981 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:07.825964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2" event={"ID":"86021cdd-6660-47bd-a5f2-1566b15e600b","Type":"ContainerDied","Data":"b620bc215313d322ded0a1fcd67f02d842551dc39e3cdd0f8a10a2f0c326e76c"} Jul 23 23:39:07.826199 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:07.826019 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b620bc215313d322ded0a1fcd67f02d842551dc39e3cdd0f8a10a2f0c326e76c" Jul 23 23:39:12.627120 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:12.627088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:17.168242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:17.168210 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2"] Jul 23 23:39:17.170638 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:17.170613 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:17.170911 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:17.170893 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8f14c64d-dg6v2"] Jul 23 23:39:17.333711 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:17.333677 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:17.580470 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:17.580399 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:17.833615 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:17.833490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:17.931906 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:17.931872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:18.179904 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:18.179822 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:18.427461 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:18.427425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:18.527507 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:18.527419 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:19.166467 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:19.166436 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86021cdd-6660-47bd-a5f2-1566b15e600b" path="/var/lib/kubelet/pods/86021cdd-6660-47bd-a5f2-1566b15e600b/volumes" Jul 23 23:39:20.922233 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:20.922201 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:21.029621 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:21.029592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:21.285520 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:21.285400 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:21.522470 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:21.522376 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:21.622736 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:21.622700 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:39:45.869152 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.869121 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4"] Jul 23 23:39:45.869583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.869536 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86021cdd-6660-47bd-a5f2-1566b15e600b" containerName="revoke-keys" Jul 23 23:39:45.869583 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.869548 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86021cdd-6660-47bd-a5f2-1566b15e600b" containerName="revoke-keys" Jul 23 23:39:45.869657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.869612 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="86021cdd-6660-47bd-a5f2-1566b15e600b" containerName="revoke-keys" Jul 23 23:39:45.872669 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.872654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" Jul 23 23:39:45.874555 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.874533 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:39:45.879644 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.879622 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4"] Jul 23 23:39:45.911783 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:45.911763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5de933-99a1-4660-b889-fbd71f889c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4\" (UID: \"3a5de933-99a1-4660-b889-fbd71f889c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" Jul 23 23:39:46.012275 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:46.012246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5de933-99a1-4660-b889-fbd71f889c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4\" (UID: \"3a5de933-99a1-4660-b889-fbd71f889c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" Jul 23 23:39:46.012866 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:46.012840 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5de933-99a1-4660-b889-fbd71f889c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4\" (UID: \"3a5de933-99a1-4660-b889-fbd71f889c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" Jul 23 23:39:46.184313 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:46.184239 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" Jul 23 23:39:46.315476 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:46.315451 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4"] Jul 23 23:39:46.318012 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:39:46.317983 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5de933_99a1_4660_b889_fbd71f889c1d.slice/crio-cd71c7ef71475281571f1ede300e2c0e0d1651b22802554afbf7722bd1d95430 WatchSource:0}: Error finding container cd71c7ef71475281571f1ede300e2c0e0d1651b22802554afbf7722bd1d95430: Status 404 returned error can't find the container with id cd71c7ef71475281571f1ede300e2c0e0d1651b22802554afbf7722bd1d95430 Jul 23 23:39:47.021114 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:47.021074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" event={"ID":"3a5de933-99a1-4660-b889-fbd71f889c1d","Type":"ContainerStarted","Data":"3a65381f324eafb8f8389ab9de6d800b42e37c76ea68f2e5aa49068eb0401745"} Jul 23 23:39:47.021502 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:47.021138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" event={"ID":"3a5de933-99a1-4660-b889-fbd71f889c1d","Type":"ContainerStarted","Data":"cd71c7ef71475281571f1ede300e2c0e0d1651b22802554afbf7722bd1d95430"} Jul 23 23:39:47.034498 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:47.034453 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" podStartSLOduration=2.034432582 podStartE2EDuration="2.034432582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:39:47.032593828 +0000 UTC m=+1610.560870994" watchObservedRunningTime="2026-07-23 23:39:47.034432582 +0000 UTC m=+1610.562709754" Jul 23 23:39:48.027458 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:48.027398 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a5de933-99a1-4660-b889-fbd71f889c1d" containerID="3a65381f324eafb8f8389ab9de6d800b42e37c76ea68f2e5aa49068eb0401745" exitCode=0 Jul 23 23:39:48.027824 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:48.027472 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" event={"ID":"3a5de933-99a1-4660-b889-fbd71f889c1d","Type":"ContainerDied","Data":"3a65381f324eafb8f8389ab9de6d800b42e37c76ea68f2e5aa49068eb0401745"} Jul 23 23:39:49.174933 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:49.174911 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" Jul 23 23:39:49.234802 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:49.234778 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5de933-99a1-4660-b889-fbd71f889c1d-maas-api-service-ca\") pod \"3a5de933-99a1-4660-b889-fbd71f889c1d\" (UID: \"3a5de933-99a1-4660-b889-fbd71f889c1d\") " Jul 23 23:39:49.235118 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:49.235094 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5de933-99a1-4660-b889-fbd71f889c1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a5de933-99a1-4660-b889-fbd71f889c1d" (UID: "3a5de933-99a1-4660-b889-fbd71f889c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:39:49.335737 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:49.335706 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5de933-99a1-4660-b889-fbd71f889c1d-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:39:50.038857 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:50.038824 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" Jul 23 23:39:50.039032 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:50.038870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4" event={"ID":"3a5de933-99a1-4660-b889-fbd71f889c1d","Type":"ContainerDied","Data":"cd71c7ef71475281571f1ede300e2c0e0d1651b22802554afbf7722bd1d95430"} Jul 23 23:39:50.039032 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:50.038946 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd71c7ef71475281571f1ede300e2c0e0d1651b22802554afbf7722bd1d95430" Jul 23 23:39:56.265283 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:39:56.265253 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:00.944202 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:00.944162 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4"] Jul 23 23:40:00.945927 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:00.945901 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:00.946532 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:00.946511 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1372ee67-tcnb4"] Jul 23 23:40:01.101232 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:01.101203 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:01.166136 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:01.166109 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5de933-99a1-4660-b889-fbd71f889c1d" path="/var/lib/kubelet/pods/3a5de933-99a1-4660-b889-fbd71f889c1d/volumes" Jul 23 23:40:01.351506 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:01.351475 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:01.601558 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:01.601463 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:01.937896 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:01.937807 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:02.043994 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:02.043960 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:04.051588 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:04.051554 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:04.149315 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:04.149277 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:04.208026 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:04.207993 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:04.391278 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:04.391250 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:04.705607 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:04.705519 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:04.805812 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:04.805778 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:29.871931 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.871898 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg"] Jul 23 23:40:29.872334 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.872288 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5de933-99a1-4660-b889-fbd71f889c1d" containerName="revoke-keys" Jul 23 23:40:29.872334 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.872299 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5de933-99a1-4660-b889-fbd71f889c1d" containerName="revoke-keys" Jul 23 23:40:29.872434 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.872374 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5de933-99a1-4660-b889-fbd71f889c1d" containerName="revoke-keys" Jul 23 23:40:29.875395 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.875380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" Jul 23 23:40:29.877112 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.877086 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:40:29.880715 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.880683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3454ffb-8651-4c0c-a13a-df458c4be158-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17320d85-lf2tg\" (UID: \"d3454ffb-8651-4c0c-a13a-df458c4be158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" Jul 23 23:40:29.881537 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.881310 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg"] Jul 23 23:40:29.981198 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.981170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3454ffb-8651-4c0c-a13a-df458c4be158-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17320d85-lf2tg\" (UID: \"d3454ffb-8651-4c0c-a13a-df458c4be158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" Jul 23 23:40:29.981774 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:29.981758 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3454ffb-8651-4c0c-a13a-df458c4be158-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-17320d85-lf2tg\" (UID: \"d3454ffb-8651-4c0c-a13a-df458c4be158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" Jul 23 23:40:30.189500 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:30.189425 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" Jul 23 23:40:30.529992 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:30.529962 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg"] Jul 23 23:40:30.532293 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:40:30.532267 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3454ffb_8651_4c0c_a13a_df458c4be158.slice/crio-8d1114be399f228a42d8f8bd11b50902abf20af6b4b0c05d31d1e86e3370df91 WatchSource:0}: Error finding container 8d1114be399f228a42d8f8bd11b50902abf20af6b4b0c05d31d1e86e3370df91: Status 404 returned error can't find the container with id 8d1114be399f228a42d8f8bd11b50902abf20af6b4b0c05d31d1e86e3370df91 Jul 23 23:40:31.246486 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:31.246458 2568 generic.go:356] "Generic (PLEG): container finished" podID="d3454ffb-8651-4c0c-a13a-df458c4be158" containerID="8dc72c19c2f008efa0962197f228546cccfa98816e4c7c9585ec6d93905d6185" exitCode=0 Jul 23 23:40:31.246785 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:31.246518 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" event={"ID":"d3454ffb-8651-4c0c-a13a-df458c4be158","Type":"ContainerDied","Data":"8dc72c19c2f008efa0962197f228546cccfa98816e4c7c9585ec6d93905d6185"} Jul 23 23:40:31.246785 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:31.246553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" event={"ID":"d3454ffb-8651-4c0c-a13a-df458c4be158","Type":"ContainerStarted","Data":"8d1114be399f228a42d8f8bd11b50902abf20af6b4b0c05d31d1e86e3370df91"} Jul 23 23:40:32.388695 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:32.388673 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" Jul 23 23:40:32.402588 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:32.402562 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3454ffb-8651-4c0c-a13a-df458c4be158-maas-api-service-ca\") pod \"d3454ffb-8651-4c0c-a13a-df458c4be158\" (UID: \"d3454ffb-8651-4c0c-a13a-df458c4be158\") " Jul 23 23:40:32.402980 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:32.402956 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3454ffb-8651-4c0c-a13a-df458c4be158-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3454ffb-8651-4c0c-a13a-df458c4be158" (UID: "d3454ffb-8651-4c0c-a13a-df458c4be158"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:40:32.503450 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:32.503348 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3454ffb-8651-4c0c-a13a-df458c4be158-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:40:33.258036 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:33.258004 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" Jul 23 23:40:33.258036 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:33.258026 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg" event={"ID":"d3454ffb-8651-4c0c-a13a-df458c4be158","Type":"ContainerDied","Data":"8d1114be399f228a42d8f8bd11b50902abf20af6b4b0c05d31d1e86e3370df91"} Jul 23 23:40:33.258230 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:33.258073 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d1114be399f228a42d8f8bd11b50902abf20af6b4b0c05d31d1e86e3370df91" Jul 23 23:40:40.279766 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:40.279736 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:44.943210 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:44.943175 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:44.955229 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:44.955179 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg"] Jul 23 23:40:44.956968 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:44.956943 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-17320d85-lf2tg"] Jul 23 23:40:45.166777 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:45.166747 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3454ffb-8651-4c0c-a13a-df458c4be158" path="/var/lib/kubelet/pods/d3454ffb-8651-4c0c-a13a-df458c4be158/volumes" Jul 23 23:40:46.193836 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:46.193805 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:46.416649 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:46.416617 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:46.653623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:46.653587 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:46.963598 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:46.963525 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:47.394799 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:47.394766 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:49.169716 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:49.169682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:49.292715 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:49.292680 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:49.438977 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:49.438887 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:49.600523 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:49.600492 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:49.921242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:49.921206 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:40:50.021863 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:40:50.021831 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:14.123138 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.123101 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq"] Jul 23 23:41:14.123537 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.123507 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3454ffb-8651-4c0c-a13a-df458c4be158" containerName="revoke-keys" Jul 23 23:41:14.123537 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.123518 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3454ffb-8651-4c0c-a13a-df458c4be158" containerName="revoke-keys" Jul 23 23:41:14.123615 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.123590 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3454ffb-8651-4c0c-a13a-df458c4be158" containerName="revoke-keys" Jul 23 23:41:14.126833 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.126817 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" Jul 23 23:41:14.128612 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.128590 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:41:14.133668 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.133639 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq"] Jul 23 23:41:14.259579 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.259550 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0fde7-4d7a-400e-9cab-82a8664a847d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq\" (UID: \"ded0fde7-4d7a-400e-9cab-82a8664a847d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" Jul 23 23:41:14.360525 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.360499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0fde7-4d7a-400e-9cab-82a8664a847d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq\" (UID: \"ded0fde7-4d7a-400e-9cab-82a8664a847d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" Jul 23 23:41:14.361136 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.361112 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0fde7-4d7a-400e-9cab-82a8664a847d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq\" (UID: \"ded0fde7-4d7a-400e-9cab-82a8664a847d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" Jul 23 23:41:14.440834 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.440782 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" Jul 23 23:41:14.579015 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:14.578988 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq"] Jul 23 23:41:14.581930 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:41:14.581902 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded0fde7_4d7a_400e_9cab_82a8664a847d.slice/crio-57c0bf2cb5806664ea9e2359adca6e076ec721d20dae7074f3924e5214848f6f WatchSource:0}: Error finding container 57c0bf2cb5806664ea9e2359adca6e076ec721d20dae7074f3924e5214848f6f: Status 404 returned error can't find the container with id 57c0bf2cb5806664ea9e2359adca6e076ec721d20dae7074f3924e5214848f6f Jul 23 23:41:15.470681 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:15.470596 2568 generic.go:356] "Generic (PLEG): container finished" podID="ded0fde7-4d7a-400e-9cab-82a8664a847d" containerID="c9109fe9e045c718e04f910d976310690129fb281e03ffaffdf566958619fcff" exitCode=0 Jul 23 23:41:15.471014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:15.470673 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" event={"ID":"ded0fde7-4d7a-400e-9cab-82a8664a847d","Type":"ContainerDied","Data":"c9109fe9e045c718e04f910d976310690129fb281e03ffaffdf566958619fcff"} Jul 23 23:41:15.471014 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:15.470725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" event={"ID":"ded0fde7-4d7a-400e-9cab-82a8664a847d","Type":"ContainerStarted","Data":"57c0bf2cb5806664ea9e2359adca6e076ec721d20dae7074f3924e5214848f6f"} Jul 23 23:41:16.612556 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:16.612527 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" Jul 23 23:41:16.779821 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:16.779745 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0fde7-4d7a-400e-9cab-82a8664a847d-maas-api-service-ca\") pod \"ded0fde7-4d7a-400e-9cab-82a8664a847d\" (UID: \"ded0fde7-4d7a-400e-9cab-82a8664a847d\") " Jul 23 23:41:16.780113 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:16.780091 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded0fde7-4d7a-400e-9cab-82a8664a847d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ded0fde7-4d7a-400e-9cab-82a8664a847d" (UID: "ded0fde7-4d7a-400e-9cab-82a8664a847d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:41:16.880480 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:16.880457 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0fde7-4d7a-400e-9cab-82a8664a847d-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:41:17.481841 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:17.481736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" event={"ID":"ded0fde7-4d7a-400e-9cab-82a8664a847d","Type":"ContainerDied","Data":"57c0bf2cb5806664ea9e2359adca6e076ec721d20dae7074f3924e5214848f6f"} Jul 23 23:41:17.481841 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:17.481793 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57c0bf2cb5806664ea9e2359adca6e076ec721d20dae7074f3924e5214848f6f" Jul 23 23:41:17.481841 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:17.481751 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq" Jul 23 23:41:24.712564 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:24.712492 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:29.192082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:29.192049 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq"] Jul 23 23:41:29.196496 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:29.196471 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7cb9a6a8-qrtpq"] Jul 23 23:41:29.196653 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:29.196591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:29.364552 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:29.364519 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:29.602024 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:29.602002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:30.096593 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:30.096563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:30.207333 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:30.207302 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:30.295835 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:30.295802 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:31.166567 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:31.166536 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded0fde7-4d7a-400e-9cab-82a8664a847d" path="/var/lib/kubelet/pods/ded0fde7-4d7a-400e-9cab-82a8664a847d/volumes" Jul 23 23:41:31.796373 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:31.796341 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:33.737638 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:33.737597 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:33.842229 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:33.842193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:37.955224 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:37.955186 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:38.061641 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:38.061605 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:38.160123 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:38.160083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:38.256080 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:38.256011 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:41:38.659623 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:41:38.659593 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:04.630791 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:04.630752 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:08.768747 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:08.768713 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:10.667021 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:10.666990 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:10.767233 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:10.767201 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:25.312332 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:25.312300 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:26.657378 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:26.657340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:32.394975 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:32.394939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:32.495967 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:32.495937 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:32.595601 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:32.595565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:43.169425 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:43.169383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:43.982315 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:43.982285 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:50.133254 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:50.133171 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:50.236649 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:50.236617 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:50.335757 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:50.335723 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:50.463400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:50.463318 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:51.245116 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:51.245086 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:51.346128 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:51.346093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:51.445553 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:51.445519 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:53.170497 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:53.170464 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:57.072319 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:57.072292 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:42:57.079934 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:57.079910 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:42:57.080868 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:57.080848 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:42:57.088340 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:57.088317 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:42:58.939382 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:58.939340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:59.039352 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:59.039323 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:42:59.615641 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:42:59.615606 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:05.761449 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:05.761328 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:05.819678 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:05.819633 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:05.998944 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:05.998905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:06.313144 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:06.313101 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:06.804939 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:06.804904 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:09.525427 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:09.525379 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:09.609689 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:09.609649 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:09.715466 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:09.715436 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:09.889809 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:09.889778 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:10.201561 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:10.201481 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:10.309917 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:10.309888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:13.977844 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:13.977813 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:14.666017 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:14.665982 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:14.764041 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:14.764004 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:14.862783 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:14.862748 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:15.113184 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:15.113153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:20.732899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:20.732857 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:20.833183 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:20.833145 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:21.099190 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.099158 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k"] Jul 23 23:43:21.099577 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.099565 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded0fde7-4d7a-400e-9cab-82a8664a847d" containerName="revoke-keys" Jul 23 23:43:21.099635 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.099579 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded0fde7-4d7a-400e-9cab-82a8664a847d" containerName="revoke-keys" Jul 23 23:43:21.099672 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.099658 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded0fde7-4d7a-400e-9cab-82a8664a847d" containerName="revoke-keys" Jul 23 23:43:21.102898 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.102882 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" Jul 23 23:43:21.104702 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.104681 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:43:21.108047 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.108023 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k"] Jul 23 23:43:21.137074 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.137043 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a2c9f-9b16-4413-a461-34684c719f52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4155fe38-9572k\" (UID: \"bc0a2c9f-9b16-4413-a461-34684c719f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" Jul 23 23:43:21.238000 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.237974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a2c9f-9b16-4413-a461-34684c719f52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4155fe38-9572k\" (UID: \"bc0a2c9f-9b16-4413-a461-34684c719f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" Jul 23 23:43:21.238550 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.238530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a2c9f-9b16-4413-a461-34684c719f52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4155fe38-9572k\" (UID: \"bc0a2c9f-9b16-4413-a461-34684c719f52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" Jul 23 23:43:21.415899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.415814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" Jul 23 23:43:21.554467 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:21.554444 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k"] Jul 23 23:43:21.557670 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:43:21.557636 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0a2c9f_9b16_4413_a461_34684c719f52.slice/crio-8dc0539e4c47b60540f828563ef84f93b96b5b90cb62c45e0c0b74b537789223 WatchSource:0}: Error finding container 8dc0539e4c47b60540f828563ef84f93b96b5b90cb62c45e0c0b74b537789223: Status 404 returned error can't find the container with id 8dc0539e4c47b60540f828563ef84f93b96b5b90cb62c45e0c0b74b537789223 Jul 23 23:43:22.123932 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:22.123901 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" event={"ID":"bc0a2c9f-9b16-4413-a461-34684c719f52","Type":"ContainerStarted","Data":"b9325abd2e29f9fa47241bf7e55418809c0036155ce8dc24d2fb41e273d2348c"} Jul 23 23:43:22.124307 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:22.123951 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" event={"ID":"bc0a2c9f-9b16-4413-a461-34684c719f52","Type":"ContainerStarted","Data":"8dc0539e4c47b60540f828563ef84f93b96b5b90cb62c45e0c0b74b537789223"} Jul 23 23:43:22.137558 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:22.137514 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" podStartSLOduration=1.137500851 podStartE2EDuration="1.137500851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:43:22.136224174 +0000 UTC m=+1825.664501342" watchObservedRunningTime="2026-07-23 23:43:22.137500851 +0000 UTC m=+1825.665778050" Jul 23 23:43:23.130863 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:23.130829 2568 generic.go:356] "Generic (PLEG): container finished" podID="bc0a2c9f-9b16-4413-a461-34684c719f52" containerID="b9325abd2e29f9fa47241bf7e55418809c0036155ce8dc24d2fb41e273d2348c" exitCode=0 Jul 23 23:43:23.131234 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:23.130910 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" event={"ID":"bc0a2c9f-9b16-4413-a461-34684c719f52","Type":"ContainerDied","Data":"b9325abd2e29f9fa47241bf7e55418809c0036155ce8dc24d2fb41e273d2348c"} Jul 23 23:43:24.269140 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:24.269114 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" Jul 23 23:43:24.362164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:24.362137 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a2c9f-9b16-4413-a461-34684c719f52-maas-api-service-ca\") pod \"bc0a2c9f-9b16-4413-a461-34684c719f52\" (UID: \"bc0a2c9f-9b16-4413-a461-34684c719f52\") " Jul 23 23:43:24.362490 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:24.362469 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0a2c9f-9b16-4413-a461-34684c719f52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc0a2c9f-9b16-4413-a461-34684c719f52" (UID: "bc0a2c9f-9b16-4413-a461-34684c719f52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:43:24.463198 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:24.463138 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0a2c9f-9b16-4413-a461-34684c719f52-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:43:25.142126 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:25.142100 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" Jul 23 23:43:25.142303 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:25.142126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k" event={"ID":"bc0a2c9f-9b16-4413-a461-34684c719f52","Type":"ContainerDied","Data":"8dc0539e4c47b60540f828563ef84f93b96b5b90cb62c45e0c0b74b537789223"} Jul 23 23:43:25.142303 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:25.142169 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dc0539e4c47b60540f828563ef84f93b96b5b90cb62c45e0c0b74b537789223" Jul 23 23:43:31.282490 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:31.282454 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:36.165123 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:36.165088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:36.175357 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:36.175333 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k"] Jul 23 23:43:36.177738 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:36.177715 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4155fe38-9572k"] Jul 23 23:43:36.560160 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:36.560071 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:37.166752 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:37.166719 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc0a2c9f-9b16-4413-a461-34684c719f52" path="/var/lib/kubelet/pods/bc0a2c9f-9b16-4413-a461-34684c719f52/volumes" Jul 23 23:43:42.115248 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:42.115216 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:42.216277 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:42.216250 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:42.391437 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:42.391335 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:42.626463 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:42.626398 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:42.765904 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:42.765805 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:42.854040 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:42.854006 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:42.971171 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:42.971143 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:48.696255 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:48.696221 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:48.796731 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:48.796686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:48.896293 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:48.896257 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:49.066659 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:49.066563 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:54.677862 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:54.677831 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:43:54.780361 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:43:54.780325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:01.581637 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:01.581601 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:01.945330 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:01.945239 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:02.125030 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:02.124992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:04.730125 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:04.730093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:04.816372 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:04.816328 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:04.882997 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:04.882953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:05.187030 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:05.186997 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:05.301072 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:05.301037 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:05.403337 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:05.403305 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:09.243507 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:09.243472 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:15.401486 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:15.401453 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:37.572651 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.572569 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr"] Jul 23 23:44:37.573163 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.572984 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0a2c9f-9b16-4413-a461-34684c719f52" containerName="revoke-keys" Jul 23 23:44:37.573163 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.572995 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0a2c9f-9b16-4413-a461-34684c719f52" containerName="revoke-keys" Jul 23 23:44:37.573163 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.573080 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc0a2c9f-9b16-4413-a461-34684c719f52" containerName="revoke-keys" Jul 23 23:44:37.576243 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.576225 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" Jul 23 23:44:37.585347 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.585287 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:44:37.591350 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.591323 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr"] Jul 23 23:44:37.691910 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.691876 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be9b636-e95d-4203-8201-42665285ee7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr\" (UID: \"6be9b636-e95d-4203-8201-42665285ee7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" Jul 23 23:44:37.792368 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.792336 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be9b636-e95d-4203-8201-42665285ee7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr\" (UID: \"6be9b636-e95d-4203-8201-42665285ee7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" Jul 23 23:44:37.792992 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.792967 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be9b636-e95d-4203-8201-42665285ee7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr\" (UID: \"6be9b636-e95d-4203-8201-42665285ee7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" Jul 23 23:44:37.896845 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:37.896747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" Jul 23 23:44:38.041064 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:38.039988 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr"] Jul 23 23:44:38.523596 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:38.523506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" event={"ID":"6be9b636-e95d-4203-8201-42665285ee7e","Type":"ContainerStarted","Data":"c150317a25b734685bc1f51923ace72d31b49df878db3700f6bff3dd1910555c"} Jul 23 23:44:38.523596 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:38.523571 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" event={"ID":"6be9b636-e95d-4203-8201-42665285ee7e","Type":"ContainerStarted","Data":"52ce47ea7cbe2712d02f386e89da0ba37bc77899cfad8a2842a87e274be9d8a4"} Jul 23 23:44:38.537673 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:38.537630 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" podStartSLOduration=1.537613011 podStartE2EDuration="1.537613011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:44:38.535789997 +0000 UTC m=+1902.064067164" watchObservedRunningTime="2026-07-23 23:44:38.537613011 +0000 UTC m=+1902.065890177" Jul 23 23:44:39.531060 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:39.531017 2568 generic.go:356] "Generic (PLEG): container finished" podID="6be9b636-e95d-4203-8201-42665285ee7e" containerID="c150317a25b734685bc1f51923ace72d31b49df878db3700f6bff3dd1910555c" exitCode=0 Jul 23 23:44:39.531454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:39.531062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" event={"ID":"6be9b636-e95d-4203-8201-42665285ee7e","Type":"ContainerDied","Data":"c150317a25b734685bc1f51923ace72d31b49df878db3700f6bff3dd1910555c"} Jul 23 23:44:40.676901 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:40.676876 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" Jul 23 23:44:40.823298 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:40.823271 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be9b636-e95d-4203-8201-42665285ee7e-maas-api-service-ca\") pod \"6be9b636-e95d-4203-8201-42665285ee7e\" (UID: \"6be9b636-e95d-4203-8201-42665285ee7e\") " Jul 23 23:44:40.823621 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:40.823598 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be9b636-e95d-4203-8201-42665285ee7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6be9b636-e95d-4203-8201-42665285ee7e" (UID: "6be9b636-e95d-4203-8201-42665285ee7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:44:40.924173 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:40.924140 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be9b636-e95d-4203-8201-42665285ee7e-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:44:41.543887 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:41.543802 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" event={"ID":"6be9b636-e95d-4203-8201-42665285ee7e","Type":"ContainerDied","Data":"52ce47ea7cbe2712d02f386e89da0ba37bc77899cfad8a2842a87e274be9d8a4"} Jul 23 23:44:41.543887 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:41.543822 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr" Jul 23 23:44:41.543887 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:41.543849 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ce47ea7cbe2712d02f386e89da0ba37bc77899cfad8a2842a87e274be9d8a4" Jul 23 23:44:53.023575 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:53.023539 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:57.644728 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:57.644681 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr"] Jul 23 23:44:57.646702 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:57.646675 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-88e3c218-8vfhr"] Jul 23 23:44:57.648420 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:57.648383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:44:59.166339 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:59.166298 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be9b636-e95d-4203-8201-42665285ee7e" path="/var/lib/kubelet/pods/6be9b636-e95d-4203-8201-42665285ee7e/volumes" Jul 23 23:44:59.974784 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:44:59.974747 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:05.052376 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:05.052346 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:42.098995 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:42.098941 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:42.334075 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:42.334039 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:47.800521 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:47.800483 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:47.854832 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:47.854801 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:48.078518 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:48.078485 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:48.330598 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:48.330495 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:49.420986 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:49.420953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:49.520608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:49.520575 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:49.620653 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:49.620576 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:50.364485 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:50.364393 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:50.465354 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:50.465322 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:50.564692 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:50.564624 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:51.976087 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:51.976043 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:55.945059 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:55.945032 2568 generic.go:356] "Generic (PLEG): container finished" podID="16456c9c-4389-4aa1-bcaa-86f1f911360e" containerID="6542d9003795aa76b0ff4af5bbfce70dcf951a3171e865805fd128a6fe4e63f0" exitCode=1 Jul 23 23:45:55.945381 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:55.945114 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" event={"ID":"16456c9c-4389-4aa1-bcaa-86f1f911360e","Type":"ContainerDied","Data":"6542d9003795aa76b0ff4af5bbfce70dcf951a3171e865805fd128a6fe4e63f0"} Jul 23 23:45:55.945451 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:55.945441 2568 scope.go:117] "RemoveContainer" containerID="6542d9003795aa76b0ff4af5bbfce70dcf951a3171e865805fd128a6fe4e63f0" Jul 23 23:45:56.952304 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:56.952271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" event={"ID":"16456c9c-4389-4aa1-bcaa-86f1f911360e","Type":"ContainerStarted","Data":"922da7a6d4383820ab50041251076577b0609bbf1616647b0dba70b7e03f1797"} Jul 23 23:45:56.952804 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:56.952559 2568 status_manager.go:444] "Container readiness changed for unknown container" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" containerID="cri-o://6542d9003795aa76b0ff4af5bbfce70dcf951a3171e865805fd128a6fe4e63f0" Jul 23 23:45:56.952804 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:56.952577 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:45:57.170930 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:57.170902 2568 scope.go:117] "RemoveContainer" containerID="3a65381f324eafb8f8389ab9de6d800b42e37c76ea68f2e5aa49068eb0401745" Jul 23 23:45:57.179082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:57.179066 2568 scope.go:117] "RemoveContainer" containerID="59d4ca57769b2065ae6a4c7c2ceb22796c1bac62d97b91b4d3c2f5ed8232ffc7" Jul 23 23:45:57.957887 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:57.957858 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:45:58.405198 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:58.405169 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:58.506613 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:58.506582 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:45:59.088991 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:45:59.088963 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:04.806697 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:04.806657 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:04.912145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:04.912095 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:05.007664 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:05.007626 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:08.964551 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:08.964520 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8bml" Jul 23 23:46:15.568682 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:15.568648 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:15.910164 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:15.910073 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:34.653455 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:34.653422 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:34.745893 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:34.745862 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:36.901073 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:36.901032 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:43.015245 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:43.015210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:43.115878 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:43.115845 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:43.394867 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:43.394836 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:43.713808 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:43.713723 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:43.962977 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:43.962942 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:46.570793 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:46.570752 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:46.679094 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:46.679060 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:46.769995 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:46.769965 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:46.995828 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:46.995740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:47.226562 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:47.226516 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:47.327270 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:47.327233 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:51.111268 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:51.111235 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:51.452944 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:51.452871 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:51.774166 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:51.774078 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:51.888115 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:51.888065 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:54.593082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:54.593042 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:54.680235 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:54.680193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:54.755994 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:54.755960 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:54.857444 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:54.857340 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:55.245120 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:55.245048 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:55.345575 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:55.345544 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:46:57.204323 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:57.204296 2568 scope.go:117] "RemoveContainer" containerID="8dc72c19c2f008efa0962197f228546cccfa98816e4c7c9585ec6d93905d6185" Jul 23 23:46:59.088120 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:46:59.088087 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:47:04.723368 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:04.723332 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:47:07.953810 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:07.953747 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:47:08.054142 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:08.054110 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:47:16.053489 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:16.053436 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:47:16.155196 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:16.155153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:47:23.518785 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:23.518710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:47:57.102680 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:57.102646 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:47:57.110289 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:57.110270 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:47:57.113466 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:57.113448 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:47:57.120712 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:57.120694 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:47:57.224596 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:47:57.224570 2568 scope.go:117] "RemoveContainer" containerID="c9109fe9e045c718e04f910d976310690129fb281e03ffaffdf566958619fcff" Jul 23 23:48:08.467362 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:08.467328 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:09.371500 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.371461 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:09.496559 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.496529 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd"] Jul 23 23:48:09.497034 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.496939 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be9b636-e95d-4203-8201-42665285ee7e" containerName="revoke-keys" Jul 23 23:48:09.497034 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.496951 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be9b636-e95d-4203-8201-42665285ee7e" containerName="revoke-keys" Jul 23 23:48:09.497034 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.497015 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be9b636-e95d-4203-8201-42665285ee7e" containerName="revoke-keys" Jul 23 23:48:09.500071 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.500051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" Jul 23 23:48:09.501907 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.501886 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:48:09.506086 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.506064 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd"] Jul 23 23:48:09.578161 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.578128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03046fac-8f7d-4760-a799-076dc47a762d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd\" (UID: \"03046fac-8f7d-4760-a799-076dc47a762d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" Jul 23 23:48:09.679310 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.679238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03046fac-8f7d-4760-a799-076dc47a762d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd\" (UID: \"03046fac-8f7d-4760-a799-076dc47a762d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" Jul 23 23:48:09.679867 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.679849 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03046fac-8f7d-4760-a799-076dc47a762d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd\" (UID: \"03046fac-8f7d-4760-a799-076dc47a762d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" Jul 23 23:48:09.813133 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.813108 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" Jul 23 23:48:09.949567 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.949442 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd"] Jul 23 23:48:09.951941 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:48:09.951914 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03046fac_8f7d_4760_a799_076dc47a762d.slice/crio-4ce69a2d320a4ceddd70ba27be2de7583ed318a41ee82b878ec661f259dbf142 WatchSource:0}: Error finding container 4ce69a2d320a4ceddd70ba27be2de7583ed318a41ee82b878ec661f259dbf142: Status 404 returned error can't find the container with id 4ce69a2d320a4ceddd70ba27be2de7583ed318a41ee82b878ec661f259dbf142 Jul 23 23:48:09.990463 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:09.990436 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:10.091200 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:10.091169 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:10.637594 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:10.637553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" event={"ID":"03046fac-8f7d-4760-a799-076dc47a762d","Type":"ContainerStarted","Data":"b1bec3e756e2b3871f1f646915296fbc4b6f60d400a90c1b495c9e3fd5fc84fd"} Jul 23 23:48:10.637989 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:10.637612 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" event={"ID":"03046fac-8f7d-4760-a799-076dc47a762d","Type":"ContainerStarted","Data":"4ce69a2d320a4ceddd70ba27be2de7583ed318a41ee82b878ec661f259dbf142"} Jul 23 23:48:10.653054 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:10.653010 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" podStartSLOduration=1.652996245 podStartE2EDuration="1.652996245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:48:10.65188834 +0000 UTC m=+2114.180165509" watchObservedRunningTime="2026-07-23 23:48:10.652996245 +0000 UTC m=+2114.181273412" Jul 23 23:48:11.644102 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:11.644059 2568 generic.go:356] "Generic (PLEG): container finished" podID="03046fac-8f7d-4760-a799-076dc47a762d" containerID="b1bec3e756e2b3871f1f646915296fbc4b6f60d400a90c1b495c9e3fd5fc84fd" exitCode=0 Jul 23 23:48:11.644559 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:11.644100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" event={"ID":"03046fac-8f7d-4760-a799-076dc47a762d","Type":"ContainerDied","Data":"b1bec3e756e2b3871f1f646915296fbc4b6f60d400a90c1b495c9e3fd5fc84fd"} Jul 23 23:48:12.791600 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:12.791580 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" Jul 23 23:48:12.904134 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:12.904060 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03046fac-8f7d-4760-a799-076dc47a762d-maas-api-service-ca\") pod \"03046fac-8f7d-4760-a799-076dc47a762d\" (UID: \"03046fac-8f7d-4760-a799-076dc47a762d\") " Jul 23 23:48:12.904390 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:12.904367 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03046fac-8f7d-4760-a799-076dc47a762d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03046fac-8f7d-4760-a799-076dc47a762d" (UID: "03046fac-8f7d-4760-a799-076dc47a762d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:48:13.005405 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:13.005378 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03046fac-8f7d-4760-a799-076dc47a762d-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:48:13.655581 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:13.655554 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" Jul 23 23:48:13.655741 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:13.655553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd" event={"ID":"03046fac-8f7d-4760-a799-076dc47a762d","Type":"ContainerDied","Data":"4ce69a2d320a4ceddd70ba27be2de7583ed318a41ee82b878ec661f259dbf142"} Jul 23 23:48:13.655741 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:13.655676 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ce69a2d320a4ceddd70ba27be2de7583ed318a41ee82b878ec661f259dbf142" Jul 23 23:48:19.901619 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:19.901589 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:24.560119 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:24.560088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:24.577112 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:24.577084 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd"] Jul 23 23:48:24.582908 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:24.582878 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c228ab-q58nd"] Jul 23 23:48:24.975420 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:24.975366 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:25.166591 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:25.166558 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03046fac-8f7d-4760-a799-076dc47a762d" path="/var/lib/kubelet/pods/03046fac-8f7d-4760-a799-076dc47a762d/volumes" Jul 23 23:48:30.858259 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:30.858223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:30.957234 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:30.957204 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:31.133996 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.133917 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:31.378112 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.378078 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:31.505283 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.505210 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x"] Jul 23 23:48:31.505622 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.505609 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03046fac-8f7d-4760-a799-076dc47a762d" containerName="revoke-keys" Jul 23 23:48:31.505671 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.505624 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="03046fac-8f7d-4760-a799-076dc47a762d" containerName="revoke-keys" Jul 23 23:48:31.505713 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.505705 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="03046fac-8f7d-4760-a799-076dc47a762d" containerName="revoke-keys" Jul 23 23:48:31.509884 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.509867 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" Jul 23 23:48:31.511580 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.511563 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:48:31.515798 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.515778 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x"] Jul 23 23:48:31.660041 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.660014 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b646575-d67b-4b7b-9424-fb413c80d149-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x\" (UID: \"3b646575-d67b-4b7b-9424-fb413c80d149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" Jul 23 23:48:31.761232 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.761157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b646575-d67b-4b7b-9424-fb413c80d149-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x\" (UID: \"3b646575-d67b-4b7b-9424-fb413c80d149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" Jul 23 23:48:31.761764 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.761744 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b646575-d67b-4b7b-9424-fb413c80d149-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x\" (UID: \"3b646575-d67b-4b7b-9424-fb413c80d149\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" Jul 23 23:48:31.824396 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.824373 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" Jul 23 23:48:31.965442 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:31.965397 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x"] Jul 23 23:48:31.968065 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:48:31.968030 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b646575_d67b_4b7b_9424_fb413c80d149.slice/crio-cf75f4a68936098075e49a243ddb6bb36100fa66e3df59a442132758a21187a4 WatchSource:0}: Error finding container cf75f4a68936098075e49a243ddb6bb36100fa66e3df59a442132758a21187a4: Status 404 returned error can't find the container with id cf75f4a68936098075e49a243ddb6bb36100fa66e3df59a442132758a21187a4 Jul 23 23:48:32.007971 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:32.007884 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:32.110298 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:32.110268 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:32.749356 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:32.749328 2568 generic.go:356] "Generic (PLEG): container finished" podID="3b646575-d67b-4b7b-9424-fb413c80d149" containerID="6a896a1fd814fc51c100ab45a07fd02390a034682362b9a6259f1291569762dc" exitCode=0 Jul 23 23:48:32.749536 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:32.749436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" event={"ID":"3b646575-d67b-4b7b-9424-fb413c80d149","Type":"ContainerDied","Data":"6a896a1fd814fc51c100ab45a07fd02390a034682362b9a6259f1291569762dc"} Jul 23 23:48:32.749536 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:32.749496 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" event={"ID":"3b646575-d67b-4b7b-9424-fb413c80d149","Type":"ContainerStarted","Data":"cf75f4a68936098075e49a243ddb6bb36100fa66e3df59a442132758a21187a4"} Jul 23 23:48:33.891697 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:33.891670 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" Jul 23 23:48:33.983825 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:33.983795 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b646575-d67b-4b7b-9424-fb413c80d149-maas-api-service-ca\") pod \"3b646575-d67b-4b7b-9424-fb413c80d149\" (UID: \"3b646575-d67b-4b7b-9424-fb413c80d149\") " Jul 23 23:48:33.984175 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:33.984147 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b646575-d67b-4b7b-9424-fb413c80d149-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b646575-d67b-4b7b-9424-fb413c80d149" (UID: "3b646575-d67b-4b7b-9424-fb413c80d149"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:48:34.085540 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:34.085502 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b646575-d67b-4b7b-9424-fb413c80d149-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:48:34.760675 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:34.760637 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" event={"ID":"3b646575-d67b-4b7b-9424-fb413c80d149","Type":"ContainerDied","Data":"cf75f4a68936098075e49a243ddb6bb36100fa66e3df59a442132758a21187a4"} Jul 23 23:48:34.760822 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:34.760691 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf75f4a68936098075e49a243ddb6bb36100fa66e3df59a442132758a21187a4" Jul 23 23:48:34.760822 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:34.760652 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x" Jul 23 23:48:41.980597 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:41.980565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:45.148966 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:45.148927 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:45.249541 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:45.249502 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:46.571047 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:46.571013 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x"] Jul 23 23:48:46.575008 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:46.574984 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:46.575147 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:46.575122 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d7e557-fs26x"] Jul 23 23:48:46.958000 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:46.957907 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:47.167250 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:47.167219 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b646575-d67b-4b7b-9424-fb413c80d149" path="/var/lib/kubelet/pods/3b646575-d67b-4b7b-9424-fb413c80d149/volumes" Jul 23 23:48:52.536601 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:52.536512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:52.637137 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:52.637105 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:52.819656 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:52.819624 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:53.043132 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:53.043077 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:53.396133 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:53.396101 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:53.700710 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:53.700640 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:53.800676 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:53.800633 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:56.213625 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:56.213591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:56.294373 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:56.294339 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:56.364169 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:56.364133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:56.497443 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:56.497335 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:56.772890 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:56.772801 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:48:56.874645 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:48:56.874612 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:00.981869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:00.981809 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:01.465062 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:01.465028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:01.784067 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:01.783981 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:02.171309 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:02.171278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:04.284333 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:04.284296 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:04.372300 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:04.372265 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:04.435130 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:04.435094 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:04.786310 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:04.786266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:04.993963 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:04.993928 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:05.095221 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:05.095193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:08.821364 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:08.821319 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:16.997210 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:16.997179 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss"] Jul 23 23:49:16.997674 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:16.997660 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b646575-d67b-4b7b-9424-fb413c80d149" containerName="revoke-keys" Jul 23 23:49:16.997719 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:16.997675 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b646575-d67b-4b7b-9424-fb413c80d149" containerName="revoke-keys" Jul 23 23:49:16.997754 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:16.997747 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b646575-d67b-4b7b-9424-fb413c80d149" containerName="revoke-keys" Jul 23 23:49:17.001090 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.001076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" Jul 23 23:49:17.002755 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.002731 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:49:17.006400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.006363 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss"] Jul 23 23:49:17.152541 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.152510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss\" (UID: \"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" Jul 23 23:49:17.253194 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.253116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss\" (UID: \"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" Jul 23 23:49:17.253713 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.253696 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss\" (UID: \"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" Jul 23 23:49:17.314242 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.314220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" Jul 23 23:49:17.516671 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.516593 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:17.616822 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.616788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:17.655879 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.655780 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss"] Jul 23 23:49:17.658813 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:49:17.658783 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a21ffc7_d2c0_4ea0_9f77_38d8ee3ade9b.slice/crio-30e18b78aa9556600be4510719861b912eeefc49e8e17a4b27b7f6f40511a776 WatchSource:0}: Error finding container 30e18b78aa9556600be4510719861b912eeefc49e8e17a4b27b7f6f40511a776: Status 404 returned error can't find the container with id 30e18b78aa9556600be4510719861b912eeefc49e8e17a4b27b7f6f40511a776 Jul 23 23:49:17.985602 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.985569 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" event={"ID":"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b","Type":"ContainerStarted","Data":"cd9d4297800b9bb1e8919f4be270cdc40018783804faac22c7decff7089543e0"} Jul 23 23:49:17.985747 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.985621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" event={"ID":"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b","Type":"ContainerStarted","Data":"30e18b78aa9556600be4510719861b912eeefc49e8e17a4b27b7f6f40511a776"} Jul 23 23:49:17.998530 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:17.998489 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" podStartSLOduration=1.998474813 podStartE2EDuration="1.998474813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:49:17.997389121 +0000 UTC m=+2181.525666287" watchObservedRunningTime="2026-07-23 23:49:17.998474813 +0000 UTC m=+2181.526751979" Jul 23 23:49:18.992616 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:18.992584 2568 generic.go:356] "Generic (PLEG): container finished" podID="1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b" containerID="cd9d4297800b9bb1e8919f4be270cdc40018783804faac22c7decff7089543e0" exitCode=0 Jul 23 23:49:18.992780 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:18.992654 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" event={"ID":"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b","Type":"ContainerDied","Data":"cd9d4297800b9bb1e8919f4be270cdc40018783804faac22c7decff7089543e0"} Jul 23 23:49:20.141815 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:20.141791 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" Jul 23 23:49:20.283590 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:20.283515 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b-maas-api-service-ca\") pod \"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b\" (UID: \"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b\") " Jul 23 23:49:20.283893 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:20.283868 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b" (UID: "1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:49:20.385006 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:20.384971 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:49:21.004330 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:21.004300 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" Jul 23 23:49:21.004529 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:21.004295 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss" event={"ID":"1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b","Type":"ContainerDied","Data":"30e18b78aa9556600be4510719861b912eeefc49e8e17a4b27b7f6f40511a776"} Jul 23 23:49:21.004529 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:21.004445 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e18b78aa9556600be4510719861b912eeefc49e8e17a4b27b7f6f40511a776" Jul 23 23:49:25.720722 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:25.720685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:25.818923 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:25.818890 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:27.508460 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:27.508405 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:27.608250 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:27.608214 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:32.760604 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:32.760558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:37.068061 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:37.068024 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:37.077430 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:37.077383 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss"] Jul 23 23:49:37.079267 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:37.079242 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ce89b1-b6qss"] Jul 23 23:49:37.166582 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:37.166556 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b" path="/var/lib/kubelet/pods/1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b/volumes" Jul 23 23:49:37.432461 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:37.432352 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:43.351711 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:43.351678 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:43.431400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:43.431370 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:43.672692 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:43.672607 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:43.874667 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:43.874630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:44.003899 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.003828 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk"] Jul 23 23:49:44.004477 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.004459 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b" containerName="revoke-keys" Jul 23 23:49:44.004524 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.004481 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b" containerName="revoke-keys" Jul 23 23:49:44.004587 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.004574 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a21ffc7-d2c0-4ea0-9f77-38d8ee3ade9b" containerName="revoke-keys" Jul 23 23:49:44.008305 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.008278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" Jul 23 23:49:44.010077 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.010054 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:49:44.011647 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.011625 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk"] Jul 23 23:49:44.084388 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.084363 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faff4e93-6412-45c9-9571-e38a94e406aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk\" (UID: \"faff4e93-6412-45c9-9571-e38a94e406aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" Jul 23 23:49:44.185705 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.185673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faff4e93-6412-45c9-9571-e38a94e406aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk\" (UID: \"faff4e93-6412-45c9-9571-e38a94e406aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" Jul 23 23:49:44.186400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.186375 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faff4e93-6412-45c9-9571-e38a94e406aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk\" (UID: \"faff4e93-6412-45c9-9571-e38a94e406aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" Jul 23 23:49:44.321364 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.321343 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" Jul 23 23:49:44.458256 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.457188 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk"] Jul 23 23:49:44.530976 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.530939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:44.631446 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:44.631318 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:45.135766 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:45.135736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" event={"ID":"faff4e93-6412-45c9-9571-e38a94e406aa","Type":"ContainerStarted","Data":"303e7a39455390e7994c58f7d3713863cb62e800e188aa4028ec8de8f2ed1ef2"} Jul 23 23:49:45.135934 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:45.135790 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" event={"ID":"faff4e93-6412-45c9-9571-e38a94e406aa","Type":"ContainerStarted","Data":"021c36024312d9d01a59738ad67cfd4641848d8c2ece33e70c0e05dedd9f90dc"} Jul 23 23:49:45.150282 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:45.150238 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" podStartSLOduration=2.150223242 podStartE2EDuration="2.150223242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:49:45.148538028 +0000 UTC m=+2208.676815187" watchObservedRunningTime="2026-07-23 23:49:45.150223242 +0000 UTC m=+2208.678500432" Jul 23 23:49:46.142844 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:46.142807 2568 generic.go:356] "Generic (PLEG): container finished" podID="faff4e93-6412-45c9-9571-e38a94e406aa" containerID="303e7a39455390e7994c58f7d3713863cb62e800e188aa4028ec8de8f2ed1ef2" exitCode=0 Jul 23 23:49:46.143230 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:46.142888 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" event={"ID":"faff4e93-6412-45c9-9571-e38a94e406aa","Type":"ContainerDied","Data":"303e7a39455390e7994c58f7d3713863cb62e800e188aa4028ec8de8f2ed1ef2"} Jul 23 23:49:47.277651 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:47.277628 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" Jul 23 23:49:47.313448 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:47.313423 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faff4e93-6412-45c9-9571-e38a94e406aa-maas-api-service-ca\") pod \"faff4e93-6412-45c9-9571-e38a94e406aa\" (UID: \"faff4e93-6412-45c9-9571-e38a94e406aa\") " Jul 23 23:49:47.313756 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:47.313732 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faff4e93-6412-45c9-9571-e38a94e406aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "faff4e93-6412-45c9-9571-e38a94e406aa" (UID: "faff4e93-6412-45c9-9571-e38a94e406aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:49:47.415072 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:47.415008 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faff4e93-6412-45c9-9571-e38a94e406aa-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:49:47.793159 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:47.793062 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:49:48.154279 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:48.154250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" event={"ID":"faff4e93-6412-45c9-9571-e38a94e406aa","Type":"ContainerDied","Data":"021c36024312d9d01a59738ad67cfd4641848d8c2ece33e70c0e05dedd9f90dc"} Jul 23 23:49:48.154279 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:48.154272 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk" Jul 23 23:49:48.154513 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:48.154296 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="021c36024312d9d01a59738ad67cfd4641848d8c2ece33e70c0e05dedd9f90dc" Jul 23 23:49:57.252061 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:57.252035 2568 scope.go:117] "RemoveContainer" containerID="b9325abd2e29f9fa47241bf7e55418809c0036155ce8dc24d2fb41e273d2348c" Jul 23 23:49:59.427322 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:49:59.427294 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:04.072517 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:04.072478 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:04.080936 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:04.080900 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk"] Jul 23 23:50:04.082716 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:04.082691 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9d0833-kqnwk"] Jul 23 23:50:04.482586 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:04.482488 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:05.166865 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:05.166833 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faff4e93-6412-45c9-9571-e38a94e406aa" path="/var/lib/kubelet/pods/faff4e93-6412-45c9-9571-e38a94e406aa/volumes" Jul 23 23:50:10.115117 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:10.115070 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:10.177306 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:10.177280 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:10.384608 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:10.384512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:10.625814 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:10.625787 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:10.998847 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:10.998813 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:11.320081 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:11.320049 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:11.419395 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:11.419366 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:12.235117 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:12.235084 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:14.674678 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:14.674648 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:14.760535 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:14.760498 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:14.824373 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:14.824342 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:14.967212 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:14.967125 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:15.239822 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:15.239742 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:15.337727 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:15.337693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:19.199603 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:19.199568 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:19.534461 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:19.534355 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:19.852339 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:19.852305 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:20.273656 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:20.273572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:22.715554 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:22.715523 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:22.799267 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:22.799237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:22.868951 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:22.868918 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:23.110953 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:23.110926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:23.275838 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:23.275807 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:23.372796 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:23.372723 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:27.268666 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:27.268636 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:36.171059 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:36.171028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:36.271321 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:36.271292 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:43.993031 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:43.993000 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:44.098374 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:44.098325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:46.427459 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:46.427423 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:50:57.271815 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:50:57.271781 2568 scope.go:117] "RemoveContainer" containerID="c150317a25b734685bc1f51923ace72d31b49df878db3700f6bff3dd1910555c" Jul 23 23:51:13.838533 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:51:13.838496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:51:35.403382 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:51:35.403350 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:51:35.506995 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:51:35.506961 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:52:20.082962 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:52:20.082873 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:52:57.137096 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:52:57.137063 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:52:57.144964 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:52:57.144941 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:52:57.147324 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:52:57.147306 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:52:57.154993 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:52:57.154976 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:53:17.221353 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:53:17.221314 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:53:18.203788 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:53:18.203757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:53:27.337830 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:53:27.337794 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:53:43.326251 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:53:43.326212 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:54:28.066277 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:54:28.066246 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:54:28.929053 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:54:28.929005 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:54:57.029721 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:54:57.029626 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:54:57.317516 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:54:57.317353 2568 scope.go:117] "RemoveContainer" containerID="6a896a1fd814fc51c100ab45a07fd02390a034682362b9a6259f1291569762dc" Jul 23 23:54:57.325928 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:54:57.325907 2568 scope.go:117] "RemoveContainer" containerID="b1bec3e756e2b3871f1f646915296fbc4b6f60d400a90c1b495c9e3fd5fc84fd" Jul 23 23:55:22.710750 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:55:22.710710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:55:36.994645 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:55:36.994608 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:55:37.095143 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:55:37.095094 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:55:44.722718 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:55:44.722659 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:55:44.823841 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:55:44.823814 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:55:57.352550 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:55:57.352521 2568 scope.go:117] "RemoveContainer" containerID="303e7a39455390e7994c58f7d3713863cb62e800e188aa4028ec8de8f2ed1ef2" Jul 23 23:55:57.360609 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:55:57.360585 2568 scope.go:117] "RemoveContainer" containerID="cd9d4297800b9bb1e8919f4be270cdc40018783804faac22c7decff7089543e0" Jul 23 23:56:06.813742 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:06.813704 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:07.713455 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:07.713400 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:08.716685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.716654 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7647-139623459584560"] Jul 23 23:56:08.717065 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.717051 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faff4e93-6412-45c9-9571-e38a94e406aa" containerName="revoke-keys" Jul 23 23:56:08.717108 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.717066 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="faff4e93-6412-45c9-9571-e38a94e406aa" containerName="revoke-keys" Jul 23 23:56:08.717159 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.717150 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="faff4e93-6412-45c9-9571-e38a94e406aa" containerName="revoke-keys" Jul 23 23:56:08.720082 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.720066 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:08.722035 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.722011 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cdk2h\"" Jul 23 23:56:08.732959 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.732933 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139623459584560"] Jul 23 23:56:08.866298 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.866260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfzp\" (UniqueName: \"kubernetes.io/projected/abff76ec-d5fb-44ff-92bd-33402fcd5a54-kube-api-access-2wfzp\") pod \"test-curl-7647-139623459584560\" (UID: \"abff76ec-d5fb-44ff-92bd-33402fcd5a54\") " pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:08.967224 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.967141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfzp\" (UniqueName: \"kubernetes.io/projected/abff76ec-d5fb-44ff-92bd-33402fcd5a54-kube-api-access-2wfzp\") pod \"test-curl-7647-139623459584560\" (UID: \"abff76ec-d5fb-44ff-92bd-33402fcd5a54\") " pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:08.974134 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:08.974107 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfzp\" (UniqueName: \"kubernetes.io/projected/abff76ec-d5fb-44ff-92bd-33402fcd5a54-kube-api-access-2wfzp\") pod \"test-curl-7647-139623459584560\" (UID: \"abff76ec-d5fb-44ff-92bd-33402fcd5a54\") " pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:09.030942 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:09.030919 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:09.370642 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:09.370618 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139623459584560"] Jul 23 23:56:09.372610 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:56:09.372584 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabff76ec_d5fb_44ff_92bd_33402fcd5a54.slice/crio-497e78e68ccf35d9b6d1a9f4ad30d1085e89b4f8eaf11558d8c2f2e0612196fb WatchSource:0}: Error finding container 497e78e68ccf35d9b6d1a9f4ad30d1085e89b4f8eaf11558d8c2f2e0612196fb: Status 404 returned error can't find the container with id 497e78e68ccf35d9b6d1a9f4ad30d1085e89b4f8eaf11558d8c2f2e0612196fb Jul 23 23:56:09.373819 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:09.373798 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 23:56:10.112307 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:10.112274 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139623459584560" event={"ID":"abff76ec-d5fb-44ff-92bd-33402fcd5a54","Type":"ContainerStarted","Data":"4a68f7d4ff5669cc49939e82c5b51a4a59bd0aa8c9f1875942c88060fd8b2fc6"} Jul 23 23:56:10.112678 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:10.112323 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139623459584560" event={"ID":"abff76ec-d5fb-44ff-92bd-33402fcd5a54","Type":"ContainerStarted","Data":"497e78e68ccf35d9b6d1a9f4ad30d1085e89b4f8eaf11558d8c2f2e0612196fb"} Jul 23 23:56:10.125155 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:10.125101 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7647-139623459584560" podStartSLOduration=1.46934592 podStartE2EDuration="2.125079912s" totalImagesPullingTime="655.733992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:56:10.123139491 +0000 UTC m=+2593.651416649" watchObservedRunningTime="2026-07-23 23:56:10.125079912 +0000 UTC m=+2593.653357081" Jul 23 23:56:10.220662 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:10.220638 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7647-139623459584560_abff76ec-d5fb-44ff-92bd-33402fcd5a54/test-curl-7647-139623459584560/0.log" Jul 23 23:56:11.118033 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:11.118001 2568 generic.go:356] "Generic (PLEG): container finished" podID="abff76ec-d5fb-44ff-92bd-33402fcd5a54" containerID="4a68f7d4ff5669cc49939e82c5b51a4a59bd0aa8c9f1875942c88060fd8b2fc6" exitCode=0 Jul 23 23:56:11.118454 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:11.118058 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139623459584560" event={"ID":"abff76ec-d5fb-44ff-92bd-33402fcd5a54","Type":"ContainerDied","Data":"4a68f7d4ff5669cc49939e82c5b51a4a59bd0aa8c9f1875942c88060fd8b2fc6"} Jul 23 23:56:12.263219 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.263196 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:12.292797 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.292773 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wfzp\" (UniqueName: \"kubernetes.io/projected/abff76ec-d5fb-44ff-92bd-33402fcd5a54-kube-api-access-2wfzp\") pod \"abff76ec-d5fb-44ff-92bd-33402fcd5a54\" (UID: \"abff76ec-d5fb-44ff-92bd-33402fcd5a54\") " Jul 23 23:56:12.293631 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.293610 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7647-139623459584560"] Jul 23 23:56:12.294805 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.294782 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abff76ec-d5fb-44ff-92bd-33402fcd5a54-kube-api-access-2wfzp" (OuterVolumeSpecName: "kube-api-access-2wfzp") pod "abff76ec-d5fb-44ff-92bd-33402fcd5a54" (UID: "abff76ec-d5fb-44ff-92bd-33402fcd5a54"). InnerVolumeSpecName "kube-api-access-2wfzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:56:12.298667 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.298634 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7647-139623459584560"] Jul 23 23:56:12.394217 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.394134 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfzp\" (UniqueName: \"kubernetes.io/projected/abff76ec-d5fb-44ff-92bd-33402fcd5a54-kube-api-access-2wfzp\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:56:12.428331 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.428304 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7647-139623459525936"] Jul 23 23:56:12.428753 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.428739 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abff76ec-d5fb-44ff-92bd-33402fcd5a54" containerName="test-curl-7647-139623459584560" Jul 23 23:56:12.428800 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.428756 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="abff76ec-d5fb-44ff-92bd-33402fcd5a54" containerName="test-curl-7647-139623459584560" Jul 23 23:56:12.428852 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.428817 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="abff76ec-d5fb-44ff-92bd-33402fcd5a54" containerName="test-curl-7647-139623459584560" Jul 23 23:56:12.431889 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.431874 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:12.434887 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.434863 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139623459525936"] Jul 23 23:56:12.494852 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.494821 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxwg\" (UniqueName: \"kubernetes.io/projected/31e887af-f21a-452c-8ce5-06871f7fcb9e-kube-api-access-vhxwg\") pod \"test-curl-7647-139623459525936\" (UID: \"31e887af-f21a-452c-8ce5-06871f7fcb9e\") " pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:12.595444 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.595369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxwg\" (UniqueName: \"kubernetes.io/projected/31e887af-f21a-452c-8ce5-06871f7fcb9e-kube-api-access-vhxwg\") pod \"test-curl-7647-139623459525936\" (UID: \"31e887af-f21a-452c-8ce5-06871f7fcb9e\") " pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:12.602657 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.602634 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxwg\" (UniqueName: \"kubernetes.io/projected/31e887af-f21a-452c-8ce5-06871f7fcb9e-kube-api-access-vhxwg\") pod \"test-curl-7647-139623459525936\" (UID: \"31e887af-f21a-452c-8ce5-06871f7fcb9e\") " pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:12.743334 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:12.743244 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:13.079118 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:13.079091 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139623459525936"] Jul 23 23:56:13.081394 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:56:13.081372 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31e887af_f21a_452c_8ce5_06871f7fcb9e.slice/crio-98034498921c97160e774af309f5514117cf0610ceb6f196e4ecf19b4f6f3478 WatchSource:0}: Error finding container 98034498921c97160e774af309f5514117cf0610ceb6f196e4ecf19b4f6f3478: Status 404 returned error can't find the container with id 98034498921c97160e774af309f5514117cf0610ceb6f196e4ecf19b4f6f3478 Jul 23 23:56:13.129270 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:13.129249 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="497e78e68ccf35d9b6d1a9f4ad30d1085e89b4f8eaf11558d8c2f2e0612196fb" Jul 23 23:56:13.129387 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:13.129254 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:13.130527 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:13.130502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139623459525936" event={"ID":"31e887af-f21a-452c-8ce5-06871f7fcb9e","Type":"ContainerStarted","Data":"98034498921c97160e774af309f5514117cf0610ceb6f196e4ecf19b4f6f3478"} Jul 23 23:56:13.139466 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:56:13.139438 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7647-139623459584560\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="abff76ec-d5fb-44ff-92bd-33402fcd5a54" pod="opendatahub/test-curl-7647-139623459584560" Jul 23 23:56:13.165485 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:13.165466 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abff76ec-d5fb-44ff-92bd-33402fcd5a54" path="/var/lib/kubelet/pods/abff76ec-d5fb-44ff-92bd-33402fcd5a54/volumes" Jul 23 23:56:14.138334 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:14.138300 2568 generic.go:356] "Generic (PLEG): container finished" podID="31e887af-f21a-452c-8ce5-06871f7fcb9e" containerID="cfda189ee9b7aeafcfabb227f59797973ee4009959e115bc4ddc445fc817ea01" exitCode=0 Jul 23 23:56:14.138788 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:14.138381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139623459525936" event={"ID":"31e887af-f21a-452c-8ce5-06871f7fcb9e","Type":"ContainerDied","Data":"cfda189ee9b7aeafcfabb227f59797973ee4009959e115bc4ddc445fc817ea01"} Jul 23 23:56:15.274499 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:15.274476 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:15.291361 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:15.291341 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7647-139623459525936_31e887af-f21a-452c-8ce5-06871f7fcb9e/test-curl-7647-139623459525936/0.log" Jul 23 23:56:15.313646 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:15.313621 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhxwg\" (UniqueName: \"kubernetes.io/projected/31e887af-f21a-452c-8ce5-06871f7fcb9e-kube-api-access-vhxwg\") pod \"31e887af-f21a-452c-8ce5-06871f7fcb9e\" (UID: \"31e887af-f21a-452c-8ce5-06871f7fcb9e\") " Jul 23 23:56:15.314318 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:15.314297 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7647-139623459525936"] Jul 23 23:56:15.315866 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:15.315838 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e887af-f21a-452c-8ce5-06871f7fcb9e-kube-api-access-vhxwg" (OuterVolumeSpecName: "kube-api-access-vhxwg") pod "31e887af-f21a-452c-8ce5-06871f7fcb9e" (UID: "31e887af-f21a-452c-8ce5-06871f7fcb9e"). InnerVolumeSpecName "kube-api-access-vhxwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:56:15.316525 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:15.316503 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7647-139623459525936"] Jul 23 23:56:15.414595 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:15.414530 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxwg\" (UniqueName: \"kubernetes.io/projected/31e887af-f21a-452c-8ce5-06871f7fcb9e-kube-api-access-vhxwg\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:56:16.149721 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:16.149691 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:16.149721 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:16.149708 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98034498921c97160e774af309f5514117cf0610ceb6f196e4ecf19b4f6f3478" Jul 23 23:56:16.158706 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:56:16.158677 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7647-139623459525936\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="31e887af-f21a-452c-8ce5-06871f7fcb9e" pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:17.163666 ip-10-0-135-101 kubenswrapper[2568]: E0723 23:56:17.163630 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7647-139623459525936\" is forbidden: User \"system:node:ip-10-0-135-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-101.ec2.internal' and this object" podUID="31e887af-f21a-452c-8ce5-06871f7fcb9e" pod="opendatahub/test-curl-7647-139623459525936" Jul 23 23:56:17.165799 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:17.165777 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e887af-f21a-452c-8ce5-06871f7fcb9e" path="/var/lib/kubelet/pods/31e887af-f21a-452c-8ce5-06871f7fcb9e/volumes" Jul 23 23:56:21.982487 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:21.982380 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7647-139623459657168"] Jul 23 23:56:21.982978 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:21.982959 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e887af-f21a-452c-8ce5-06871f7fcb9e" containerName="test-curl-7647-139623459525936" Jul 23 23:56:21.983065 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:21.982981 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e887af-f21a-452c-8ce5-06871f7fcb9e" containerName="test-curl-7647-139623459525936" Jul 23 23:56:21.983117 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:21.983102 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e887af-f21a-452c-8ce5-06871f7fcb9e" containerName="test-curl-7647-139623459525936" Jul 23 23:56:21.986276 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:21.986257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459657168" Jul 23 23:56:21.988313 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:21.988294 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cdk2h\"" Jul 23 23:56:21.989161 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:21.989138 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139623459657168"] Jul 23 23:56:22.060741 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:22.060712 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzmd\" (UniqueName: \"kubernetes.io/projected/ac9a2bdd-f487-41ac-a9a0-7b08597e281d-kube-api-access-hfzmd\") pod \"test-curl-7647-139623459657168\" (UID: \"ac9a2bdd-f487-41ac-a9a0-7b08597e281d\") " pod="opendatahub/test-curl-7647-139623459657168" Jul 23 23:56:22.161061 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:22.161030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzmd\" (UniqueName: \"kubernetes.io/projected/ac9a2bdd-f487-41ac-a9a0-7b08597e281d-kube-api-access-hfzmd\") pod \"test-curl-7647-139623459657168\" (UID: \"ac9a2bdd-f487-41ac-a9a0-7b08597e281d\") " pod="opendatahub/test-curl-7647-139623459657168" Jul 23 23:56:22.167980 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:22.167956 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzmd\" (UniqueName: \"kubernetes.io/projected/ac9a2bdd-f487-41ac-a9a0-7b08597e281d-kube-api-access-hfzmd\") pod \"test-curl-7647-139623459657168\" (UID: \"ac9a2bdd-f487-41ac-a9a0-7b08597e281d\") " pod="opendatahub/test-curl-7647-139623459657168" Jul 23 23:56:22.297888 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:22.297822 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459657168" Jul 23 23:56:22.429299 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:22.429258 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-139623459657168"] Jul 23 23:56:22.434029 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:56:22.433986 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac9a2bdd_f487_41ac_a9a0_7b08597e281d.slice/crio-8ed7fd88cca525d65f7760391d3fc2d2149e154552adad78b8ea399df1387af4 WatchSource:0}: Error finding container 8ed7fd88cca525d65f7760391d3fc2d2149e154552adad78b8ea399df1387af4: Status 404 returned error can't find the container with id 8ed7fd88cca525d65f7760391d3fc2d2149e154552adad78b8ea399df1387af4 Jul 23 23:56:23.186754 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:23.186722 2568 generic.go:356] "Generic (PLEG): container finished" podID="ac9a2bdd-f487-41ac-a9a0-7b08597e281d" containerID="e0e378288230bbe4a847ef89ce92d799a203fd54e07cd5ffe2c1e4aaa15e1f5f" exitCode=0 Jul 23 23:56:23.187058 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:23.186788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139623459657168" event={"ID":"ac9a2bdd-f487-41ac-a9a0-7b08597e281d","Type":"ContainerDied","Data":"e0e378288230bbe4a847ef89ce92d799a203fd54e07cd5ffe2c1e4aaa15e1f5f"} Jul 23 23:56:23.187058 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:23.186822 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-139623459657168" event={"ID":"ac9a2bdd-f487-41ac-a9a0-7b08597e281d","Type":"ContainerStarted","Data":"8ed7fd88cca525d65f7760391d3fc2d2149e154552adad78b8ea399df1387af4"} Jul 23 23:56:24.329968 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:24.329946 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459657168" Jul 23 23:56:24.348961 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:24.348937 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7647-139623459657168_ac9a2bdd-f487-41ac-a9a0-7b08597e281d/test-curl-7647-139623459657168/0.log" Jul 23 23:56:24.371062 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:24.371034 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7647-139623459657168"] Jul 23 23:56:24.373585 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:24.373564 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7647-139623459657168"] Jul 23 23:56:24.377697 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:24.377679 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfzmd\" (UniqueName: \"kubernetes.io/projected/ac9a2bdd-f487-41ac-a9a0-7b08597e281d-kube-api-access-hfzmd\") pod \"ac9a2bdd-f487-41ac-a9a0-7b08597e281d\" (UID: \"ac9a2bdd-f487-41ac-a9a0-7b08597e281d\") " Jul 23 23:56:24.379605 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:24.379582 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9a2bdd-f487-41ac-a9a0-7b08597e281d-kube-api-access-hfzmd" (OuterVolumeSpecName: "kube-api-access-hfzmd") pod "ac9a2bdd-f487-41ac-a9a0-7b08597e281d" (UID: "ac9a2bdd-f487-41ac-a9a0-7b08597e281d"). InnerVolumeSpecName "kube-api-access-hfzmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 23:56:24.478563 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:24.478507 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfzmd\" (UniqueName: \"kubernetes.io/projected/ac9a2bdd-f487-41ac-a9a0-7b08597e281d-kube-api-access-hfzmd\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:56:25.167050 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:25.167006 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac9a2bdd-f487-41ac-a9a0-7b08597e281d" path="/var/lib/kubelet/pods/ac9a2bdd-f487-41ac-a9a0-7b08597e281d/volumes" Jul 23 23:56:25.198503 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:25.198483 2568 scope.go:117] "RemoveContainer" containerID="e0e378288230bbe4a847ef89ce92d799a203fd54e07cd5ffe2c1e4aaa15e1f5f" Jul 23 23:56:25.198641 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:25.198512 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-139623459657168" Jul 23 23:56:27.086757 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.086718 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8"] Jul 23 23:56:27.087236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.087103 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9a2bdd-f487-41ac-a9a0-7b08597e281d" containerName="test-curl-7647-139623459657168" Jul 23 23:56:27.087236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.087114 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9a2bdd-f487-41ac-a9a0-7b08597e281d" containerName="test-curl-7647-139623459657168" Jul 23 23:56:27.087236 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.087213 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9a2bdd-f487-41ac-a9a0-7b08597e281d" containerName="test-curl-7647-139623459657168" Jul 23 23:56:27.091621 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.091598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" Jul 23 23:56:27.093344 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.093320 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:56:27.096587 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.096561 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8"] Jul 23 23:56:27.100071 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.100046 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184ed0bb-9101-439f-9013-ac43fca60cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-867102-d9sx8\" (UID: \"184ed0bb-9101-439f-9013-ac43fca60cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" Jul 23 23:56:27.201351 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.201328 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184ed0bb-9101-439f-9013-ac43fca60cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-867102-d9sx8\" (UID: \"184ed0bb-9101-439f-9013-ac43fca60cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" Jul 23 23:56:27.201998 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.201978 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184ed0bb-9101-439f-9013-ac43fca60cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-867102-d9sx8\" (UID: \"184ed0bb-9101-439f-9013-ac43fca60cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" Jul 23 23:56:27.404579 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.404494 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" Jul 23 23:56:27.544675 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.544651 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8"] Jul 23 23:56:27.547470 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:56:27.547402 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184ed0bb_9101_439f_9013_ac43fca60cff.slice/crio-679796ba42aad25019211896c7cc8167c49c2d6b6340b7ad6c875b3de8ed2904 WatchSource:0}: Error finding container 679796ba42aad25019211896c7cc8167c49c2d6b6340b7ad6c875b3de8ed2904: Status 404 returned error can't find the container with id 679796ba42aad25019211896c7cc8167c49c2d6b6340b7ad6c875b3de8ed2904 Jul 23 23:56:27.581471 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.581425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:27.682499 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:27.682431 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:28.218247 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:28.218204 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" event={"ID":"184ed0bb-9101-439f-9013-ac43fca60cff","Type":"ContainerStarted","Data":"6e8f0247599239e41272bc45960d33b1e14770325caa0835f28f930ef9a9f815"} Jul 23 23:56:28.218694 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:28.218276 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" event={"ID":"184ed0bb-9101-439f-9013-ac43fca60cff","Type":"ContainerStarted","Data":"679796ba42aad25019211896c7cc8167c49c2d6b6340b7ad6c875b3de8ed2904"} Jul 23 23:56:28.231285 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:28.231222 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" podStartSLOduration=1.231200106 podStartE2EDuration="1.231200106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:56:28.230385198 +0000 UTC m=+2611.758662367" watchObservedRunningTime="2026-07-23 23:56:28.231200106 +0000 UTC m=+2611.759477275" Jul 23 23:56:29.224603 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:29.224568 2568 generic.go:356] "Generic (PLEG): container finished" podID="184ed0bb-9101-439f-9013-ac43fca60cff" containerID="6e8f0247599239e41272bc45960d33b1e14770325caa0835f28f930ef9a9f815" exitCode=0 Jul 23 23:56:29.225086 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:29.224663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" event={"ID":"184ed0bb-9101-439f-9013-ac43fca60cff","Type":"ContainerDied","Data":"6e8f0247599239e41272bc45960d33b1e14770325caa0835f28f930ef9a9f815"} Jul 23 23:56:30.381991 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:30.381967 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" Jul 23 23:56:30.428389 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:30.428361 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184ed0bb-9101-439f-9013-ac43fca60cff-maas-api-service-ca\") pod \"184ed0bb-9101-439f-9013-ac43fca60cff\" (UID: \"184ed0bb-9101-439f-9013-ac43fca60cff\") " Jul 23 23:56:30.428705 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:30.428680 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184ed0bb-9101-439f-9013-ac43fca60cff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "184ed0bb-9101-439f-9013-ac43fca60cff" (UID: "184ed0bb-9101-439f-9013-ac43fca60cff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:56:30.529675 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:30.529612 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/184ed0bb-9101-439f-9013-ac43fca60cff-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:56:31.236253 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:31.236225 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" Jul 23 23:56:31.236388 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:31.236262 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8" event={"ID":"184ed0bb-9101-439f-9013-ac43fca60cff","Type":"ContainerDied","Data":"679796ba42aad25019211896c7cc8167c49c2d6b6340b7ad6c875b3de8ed2904"} Jul 23 23:56:31.236388 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:31.236304 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679796ba42aad25019211896c7cc8167c49c2d6b6340b7ad6c875b3de8ed2904" Jul 23 23:56:36.241570 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:36.241532 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:36.342546 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:36.342512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:37.633564 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:37.633527 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:37.732548 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:37.732512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:37.864686 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:37.864650 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:42.160940 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:42.160901 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:42.176291 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:42.176263 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8"] Jul 23 23:56:42.179166 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:42.179144 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-867102-d9sx8"] Jul 23 23:56:42.591944 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:42.591915 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:43.166471 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:43.166437 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="184ed0bb-9101-439f-9013-ac43fca60cff" path="/var/lib/kubelet/pods/184ed0bb-9101-439f-9013-ac43fca60cff/volumes" Jul 23 23:56:48.376180 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:48.376146 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:48.427239 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:48.427212 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:48.652375 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:48.652292 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:48.909898 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:48.909821 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:49.039523 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.039493 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz"] Jul 23 23:56:49.039921 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.039907 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="184ed0bb-9101-439f-9013-ac43fca60cff" containerName="revoke-keys" Jul 23 23:56:49.039966 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.039924 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="184ed0bb-9101-439f-9013-ac43fca60cff" containerName="revoke-keys" Jul 23 23:56:49.040002 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.039989 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="184ed0bb-9101-439f-9013-ac43fca60cff" containerName="revoke-keys" Jul 23 23:56:49.045127 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.045107 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" Jul 23 23:56:49.046968 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.046945 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzd6x\"" Jul 23 23:56:49.049478 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.049450 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz"] Jul 23 23:56:49.193051 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.192969 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f1a03f-e898-4972-bdfe-1f758ac6f347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz\" (UID: \"87f1a03f-e898-4972-bdfe-1f758ac6f347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" Jul 23 23:56:49.293594 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.293567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f1a03f-e898-4972-bdfe-1f758ac6f347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz\" (UID: \"87f1a03f-e898-4972-bdfe-1f758ac6f347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" Jul 23 23:56:49.294145 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.294127 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f1a03f-e898-4972-bdfe-1f758ac6f347-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz\" (UID: \"87f1a03f-e898-4972-bdfe-1f758ac6f347\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" Jul 23 23:56:49.358859 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.358835 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" Jul 23 23:56:49.630206 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.630173 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:49.703792 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.703734 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz"] Jul 23 23:56:49.705563 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:49.705536 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:56:49.706437 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:56:49.706388 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f1a03f_e898_4972_bdfe_1f758ac6f347.slice/crio-621863a27c37f7940ee3cee773be6edc8ee8dac3ff0c7bdc90c42cc7f2629359 WatchSource:0}: Error finding container 621863a27c37f7940ee3cee773be6edc8ee8dac3ff0c7bdc90c42cc7f2629359: Status 404 returned error can't find the container with id 621863a27c37f7940ee3cee773be6edc8ee8dac3ff0c7bdc90c42cc7f2629359 Jul 23 23:56:50.335663 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:50.335625 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" event={"ID":"87f1a03f-e898-4972-bdfe-1f758ac6f347","Type":"ContainerStarted","Data":"4c0cc3c889992879a503afe2cc4038731595ac4e013f3cfa4cfbb389b95d06f2"} Jul 23 23:56:50.335858 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:50.335690 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" event={"ID":"87f1a03f-e898-4972-bdfe-1f758ac6f347","Type":"ContainerStarted","Data":"621863a27c37f7940ee3cee773be6edc8ee8dac3ff0c7bdc90c42cc7f2629359"} Jul 23 23:56:50.348758 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:50.348707 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" podStartSLOduration=1.3486934210000001 podStartE2EDuration="1.348693421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 23:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:56:50.347403229 +0000 UTC m=+2633.875680395" watchObservedRunningTime="2026-07-23 23:56:50.348693421 +0000 UTC m=+2633.876970588" Jul 23 23:56:51.343289 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:51.343251 2568 generic.go:356] "Generic (PLEG): container finished" podID="87f1a03f-e898-4972-bdfe-1f758ac6f347" containerID="4c0cc3c889992879a503afe2cc4038731595ac4e013f3cfa4cfbb389b95d06f2" exitCode=0 Jul 23 23:56:51.343699 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:51.343287 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" event={"ID":"87f1a03f-e898-4972-bdfe-1f758ac6f347","Type":"ContainerDied","Data":"4c0cc3c889992879a503afe2cc4038731595ac4e013f3cfa4cfbb389b95d06f2"} Jul 23 23:56:52.494337 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:52.494317 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" Jul 23 23:56:52.621633 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:52.621550 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f1a03f-e898-4972-bdfe-1f758ac6f347-maas-api-service-ca\") pod \"87f1a03f-e898-4972-bdfe-1f758ac6f347\" (UID: \"87f1a03f-e898-4972-bdfe-1f758ac6f347\") " Jul 23 23:56:52.621882 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:52.621859 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87f1a03f-e898-4972-bdfe-1f758ac6f347-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87f1a03f-e898-4972-bdfe-1f758ac6f347" (UID: "87f1a03f-e898-4972-bdfe-1f758ac6f347"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 23:56:52.722548 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:52.722521 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87f1a03f-e898-4972-bdfe-1f758ac6f347-maas-api-service-ca\") on node \"ip-10-0-135-101.ec2.internal\" DevicePath \"\"" Jul 23 23:56:53.358308 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:53.358282 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" Jul 23 23:56:53.358517 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:53.358326 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz" event={"ID":"87f1a03f-e898-4972-bdfe-1f758ac6f347","Type":"ContainerDied","Data":"621863a27c37f7940ee3cee773be6edc8ee8dac3ff0c7bdc90c42cc7f2629359"} Jul 23 23:56:53.358517 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:53.358374 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="621863a27c37f7940ee3cee773be6edc8ee8dac3ff0c7bdc90c42cc7f2629359" Jul 23 23:56:59.586020 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:56:59.585987 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:04.108670 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:04.108638 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:04.115350 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:04.115323 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz"] Jul 23 23:57:04.117228 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:04.117208 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fc333b-jdfxz"] Jul 23 23:57:04.506298 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:04.506188 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:05.166624 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:05.166593 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87f1a03f-e898-4972-bdfe-1f758ac6f347" path="/var/lib/kubelet/pods/87f1a03f-e898-4972-bdfe-1f758ac6f347/volumes" Jul 23 23:57:10.261457 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:10.261403 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:10.362545 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:10.362513 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:10.541352 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:10.541273 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:10.795480 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:10.795376 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:16.803621 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:16.803589 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rv9m5_eb593bef-e501-44e6-9ffb-7b826a62447b/manager/0.log" Jul 23 23:57:17.056435 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:17.056351 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79fccc4f95-h6rn8_69a21b07-4fe7-48e7-a626-7d492ea5066a/manager/0.log" Jul 23 23:57:17.177685 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:17.177658 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-74w8s_5a405e7f-594a-4056-a8b6-2e87ab22a26a/server/0.log" Jul 23 23:57:17.297065 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:17.297039 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lz99j_b2a54924-975a-4580-866d-7f3dcf714b3f/manager/0.log" Jul 23 23:57:17.690878 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:17.690849 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd4b8775f-rdv7g_6625b413-418a-45b5-b19c-6ee7b881f9a0/copy-manifests/0.log" Jul 23 23:57:17.696404 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:17.696371 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd4b8775f-rdv7g_6625b413-418a-45b5-b19c-6ee7b881f9a0/manager/0.log" Jul 23 23:57:18.402217 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:18.402178 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-sdmv5_1e62999c-0ddd-410b-817a-3b744196df42/postgres/0.log" Jul 23 23:57:19.860867 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:19.860827 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-q4plc_f28034e7-27ea-4020-a0a9-5f7ad5a5fc16/manager/0.log" Jul 23 23:57:19.979400 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:19.979375 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bwv2v_d1b4af67-c153-43e0-ba31-aa3a47f5d737/manager/0.log" Jul 23 23:57:20.211395 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:20.211318 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fmbzg_390e0cea-0644-44e6-9865-2419f2aca459/registry-server/0.log" Jul 23 23:57:20.342491 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:20.342459 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cd588_f7cee4f6-a438-48d3-9d01-3fdb299ecbf1/manager/0.log" Jul 23 23:57:20.602610 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:20.602575 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q8bml_16456c9c-4389-4aa1-bcaa-86f1f911360e/manager/1.log" Jul 23 23:57:21.326895 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:21.326864 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b54995f6-wzbds_fc357fcf-7057-4f2a-95f2-48e85205127e/kube-auth-proxy/0.log" Jul 23 23:57:21.713701 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:21.713630 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s5wdx_27c8392a-402b-4436-950f-ab76923eb373/payload-processing/0.log" Jul 23 23:57:21.830043 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:21.830012 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c6465647-5pnmv_7a47b482-7b1c-4c44-aa8c-23e5b0d262c6/router/0.log" Jul 23 23:57:22.186344 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:22.186304 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-wjd54_348646c1-964d-4bb5-9a8d-a372e9edb1ba/main/0.log" Jul 23 23:57:22.548013 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:22.547937 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8vhxj_20022134-2259-4f38-a852-fab522a79ffa/main/0.log" Jul 23 23:57:22.661016 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:22.660987 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-r527m_529f04f6-49bd-41c3-8187-74d669a4e2da/main/0.log" Jul 23 23:57:28.744835 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:28.744798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:28.809804 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:28.809775 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:31.461017 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:31.460991 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hc8bv_d90782c3-1c42-4715-804d-63e1d26969f2/global-pull-secret-syncer/0.log" Jul 23 23:57:31.541035 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:31.541005 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" Jul 23 23:57:31.685015 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:31.684990 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-101.ec2.internal_ed436c2c06eb421aaa817d4e8a2cf780/haproxy/0.log" Jul 23 23:57:37.382891 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:37.382862 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-q4plc_f28034e7-27ea-4020-a0a9-5f7ad5a5fc16/manager/0.log" Jul 23 23:57:37.404211 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:37.404188 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bwv2v_d1b4af67-c153-43e0-ba31-aa3a47f5d737/manager/0.log" Jul 23 23:57:37.470007 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:37.469984 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fmbzg_390e0cea-0644-44e6-9865-2419f2aca459/registry-server/0.log" Jul 23 23:57:37.587172 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:37.587140 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cd588_f7cee4f6-a438-48d3-9d01-3fdb299ecbf1/manager/0.log" Jul 23 23:57:37.661606 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:37.661519 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q8bml_16456c9c-4389-4aa1-bcaa-86f1f911360e/manager/1.log" Jul 23 23:57:37.677245 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:37.677222 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-q8bml_16456c9c-4389-4aa1-bcaa-86f1f911360e/manager/0.log" Jul 23 23:57:39.476279 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:39.476249 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-prmzf_b4ef8031-4d56-4616-932e-73bd80525891/cluster-monitoring-operator/0.log" Jul 23 23:57:39.620772 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:39.620747 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59zzq_02c415ac-93bf-4462-a557-5eb4f6543eba/node-exporter/0.log" Jul 23 23:57:39.638458 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:39.638435 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59zzq_02c415ac-93bf-4462-a557-5eb4f6543eba/kube-rbac-proxy/0.log" Jul 23 23:57:39.657471 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:39.657449 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-59zzq_02c415ac-93bf-4462-a557-5eb4f6543eba/init-textfile/0.log" Jul 23 23:57:41.454688 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.454663 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-2cb84_893999eb-09a3-46c0-9f3e-8c7d3e008682/networking-console-plugin/0.log" Jul 23 23:57:41.792004 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.791930 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w"] Jul 23 23:57:41.792329 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.792317 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87f1a03f-e898-4972-bdfe-1f758ac6f347" containerName="revoke-keys" Jul 23 23:57:41.792376 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.792330 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="87f1a03f-e898-4972-bdfe-1f758ac6f347" containerName="revoke-keys" Jul 23 23:57:41.792440 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.792434 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="87f1a03f-e898-4972-bdfe-1f758ac6f347" containerName="revoke-keys" Jul 23 23:57:41.795686 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.795667 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.797336 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.797316 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gm46q\"/\"openshift-service-ca.crt\"" Jul 23 23:57:41.797448 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.797316 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gm46q\"/\"kube-root-ca.crt\"" Jul 23 23:57:41.797654 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.797640 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gm46q\"/\"default-dockercfg-m5j7c\"" Jul 23 23:57:41.801426 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.801384 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w"] Jul 23 23:57:41.829776 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.829753 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-sys\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.829878 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.829789 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbbp\" (UniqueName: \"kubernetes.io/projected/b769d0e5-a441-404a-aa09-16c9085e7397-kube-api-access-fwbbp\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.829878 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.829808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-proc\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.829878 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.829826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-podres\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.829991 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.829935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-lib-modules\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.930736 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-sys\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.930869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbbp\" (UniqueName: \"kubernetes.io/projected/b769d0e5-a441-404a-aa09-16c9085e7397-kube-api-access-fwbbp\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.930869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930763 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-proc\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.930869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-podres\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.930869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930822 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-sys\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.930869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-lib-modules\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.930869 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-proc\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.931158 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930910 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-podres\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.931158 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.930970 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b769d0e5-a441-404a-aa09-16c9085e7397-lib-modules\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:41.938008 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:41.937990 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbbp\" (UniqueName: \"kubernetes.io/projected/b769d0e5-a441-404a-aa09-16c9085e7397-kube-api-access-fwbbp\") pod \"perf-node-gather-daemonset-7768w\" (UID: \"b769d0e5-a441-404a-aa09-16c9085e7397\") " pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:42.105246 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.105226 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:57:42.105934 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.105921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:42.111937 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.111918 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/2.log" Jul 23 23:57:42.237120 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.237098 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w"] Jul 23 23:57:42.239285 ip-10-0-135-101 kubenswrapper[2568]: W0723 23:57:42.239260 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb769d0e5_a441_404a_aa09_16c9085e7397.slice/crio-c7435625c13ca132b32dc353a30900d7dc97aec45433e9b1cf10e9359821b885 WatchSource:0}: Error finding container c7435625c13ca132b32dc353a30900d7dc97aec45433e9b1cf10e9359821b885: Status 404 returned error can't find the container with id c7435625c13ca132b32dc353a30900d7dc97aec45433e9b1cf10e9359821b885 Jul 23 23:57:42.613425 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.613380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" event={"ID":"b769d0e5-a441-404a-aa09-16c9085e7397","Type":"ContainerStarted","Data":"6d536b14765f99e6e8ffbce328dd54921cb22d157557124086e6d94e466f4fad"} Jul 23 23:57:42.613811 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.613460 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:42.613811 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.613481 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" event={"ID":"b769d0e5-a441-404a-aa09-16c9085e7397","Type":"ContainerStarted","Data":"c7435625c13ca132b32dc353a30900d7dc97aec45433e9b1cf10e9359821b885"} Jul 23 23:57:42.634585 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.629911 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" podStartSLOduration=1.6298922660000001 podStartE2EDuration="1.629892266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 23:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 23:57:42.626564018 +0000 UTC m=+2686.154841182" watchObservedRunningTime="2026-07-23 23:57:42.629892266 +0000 UTC m=+2686.158169434" Jul 23 23:57:42.661717 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:42.661696 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-2ms2f_aad21b2f-e0d8-4aa4-bd2c-834e23b6f3ba/download-server/0.log" Jul 23 23:57:43.260225 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:43.260194 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-mtjx8_30a1c6ad-fda1-4caa-b434-5ed525f489e2/volume-data-source-validator/0.log" Jul 23 23:57:44.238186 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:44.238155 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8k2pn_fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f/dns/0.log" Jul 23 23:57:44.257029 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:44.257010 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8k2pn_fb3e9ebe-aab7-4b0d-b12e-c6a9b5de481f/kube-rbac-proxy/0.log" Jul 23 23:57:44.413656 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:44.413634 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wz6vs_63158caf-bd4e-4285-9544-e5abb2424302/dns-node-resolver/0.log" Jul 23 23:57:44.959803 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:44.959773 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nlggj_4be791d5-2577-4234-9289-6c870b08b7f8/node-ca/0.log" Jul 23 23:57:45.600835 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:45.600797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:46.229554 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:46.229528 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b54995f6-wzbds_fc357fcf-7057-4f2a-95f2-48e85205127e/kube-auth-proxy/0.log" Jul 23 23:57:46.373483 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:46.373459 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s5wdx_27c8392a-402b-4436-950f-ab76923eb373/payload-processing/0.log" Jul 23 23:57:46.394906 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:46.394888 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c6465647-5pnmv_7a47b482-7b1c-4c44-aa8c-23e5b0d262c6/router/0.log" Jul 23 23:57:47.056522 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:47.056482 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-27mq6_ff576690-faba-4356-9a7c-a66d0e79daf7/serve-healthcheck-canary/0.log" Jul 23 23:57:47.595488 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:47.595460 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-f9q5n_888f0508-837a-4dd8-bd36-e5e0c752d917/insights-operator/0.log" Jul 23 23:57:47.595985 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:47.595969 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-f9q5n_888f0508-837a-4dd8-bd36-e5e0c752d917/insights-operator/1.log" Jul 23 23:57:47.673476 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:47.673449 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p6zhr_79d0ade1-d285-489d-a7ad-0a1b3afd4943/kube-rbac-proxy/0.log" Jul 23 23:57:47.691472 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:47.691449 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p6zhr_79d0ade1-d285-489d-a7ad-0a1b3afd4943/exporter/0.log" Jul 23 23:57:47.709176 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:47.709130 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p6zhr_79d0ade1-d285-489d-a7ad-0a1b3afd4943/extractor/0.log" Jul 23 23:57:48.628925 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:48.628889 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gm46q/perf-node-gather-daemonset-7768w" Jul 23 23:57:49.788385 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:49.788309 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rv9m5_eb593bef-e501-44e6-9ffb-7b826a62447b/manager/0.log" Jul 23 23:57:50.268541 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:50.268515 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79fccc4f95-h6rn8_69a21b07-4fe7-48e7-a626-7d492ea5066a/manager/0.log" Jul 23 23:57:50.291396 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:50.291364 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-74w8s_5a405e7f-594a-4056-a8b6-2e87ab22a26a/server/0.log" Jul 23 23:57:50.353605 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:50.353578 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lz99j_b2a54924-975a-4580-866d-7f3dcf714b3f/manager/0.log" Jul 23 23:57:50.559844 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:50.559771 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd4b8775f-rdv7g_6625b413-418a-45b5-b19c-6ee7b881f9a0/manager/0.log" Jul 23 23:57:50.577740 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:50.577716 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd4b8775f-rdv7g_6625b413-418a-45b5-b19c-6ee7b881f9a0/copy-manifests/0.log" Jul 23 23:57:52.280238 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:52.280161 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b54995f6-wzbds_fc357fcf-7057-4f2a-95f2-48e85205127e/kube-auth-proxy/0.log" Jul 23 23:57:52.400347 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:52.400315 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s5wdx_27c8392a-402b-4436-950f-ab76923eb373/payload-processing/0.log" Jul 23 23:57:52.416113 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:52.416093 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c6465647-5pnmv_7a47b482-7b1c-4c44-aa8c-23e5b0d262c6/router/0.log" Jul 23 23:57:53.645452 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:53.645367 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7n678_e93b0ed8-c722-414d-8752-4dac337b6739/konnectivity-agent/0.log" limit=1024 Jul 23 23:57:53.807143 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:53.807118 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b54995f6-wzbds_fc357fcf-7057-4f2a-95f2-48e85205127e/kube-auth-proxy/0.log" Jul 23 23:57:53.935027 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:53.934959 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-s5wdx_27c8392a-402b-4436-950f-ab76923eb373/payload-processing/0.log" Jul 23 23:57:53.951690 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:53.951660 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-78c6465647-5pnmv_7a47b482-7b1c-4c44-aa8c-23e5b0d262c6/router/0.log" Jul 23 23:57:56.738711 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:56.738687 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-lrcjp_be93e488-fb0d-4505-8474-8a7763712fdd/migrator/0.log" Jul 23 23:57:56.759992 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:56.759967 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-lrcjp_be93e488-fb0d-4505-8474-8a7763712fdd/graceful-termination/0.log" Jul 23 23:57:57.106863 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:57.106828 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl_c7b7d3d4-4175-4601-862b-23b9048ab96b/kube-storage-version-migrator-operator/1.log" Jul 23 23:57:57.107794 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:57.107774 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-w9svl_c7b7d3d4-4175-4601-862b-23b9048ab96b/kube-storage-version-migrator-operator/0.log" Jul 23 23:57:57.169761 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:57.169734 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:57:57.177420 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:57.177388 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:57:57.180141 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:57.180123 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tg2ns_2038a8f5-2d28-4cb5-a57d-c026505bd5fc/console-operator/1.log" Jul 23 23:57:57.187598 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:57.187583 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zb8ln_afe83ceb-ece6-4972-b6ef-cf1671100c7b/ovn-acl-logging/0.log" Jul 23 23:57:58.067005 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.066978 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5vxfr_03fd2de7-bc46-4700-aa22-e765108fe445/kube-multus/0.log" Jul 23 23:57:58.216393 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.216369 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stvz9_47c58c88-b124-4e13-8596-79c1031522fa/kube-multus-additional-cni-plugins/0.log" Jul 23 23:57:58.234985 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.234960 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stvz9_47c58c88-b124-4e13-8596-79c1031522fa/egress-router-binary-copy/0.log" Jul 23 23:57:58.252683 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.252662 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stvz9_47c58c88-b124-4e13-8596-79c1031522fa/cni-plugins/0.log" Jul 23 23:57:58.270429 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.270392 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stvz9_47c58c88-b124-4e13-8596-79c1031522fa/bond-cni-plugin/0.log" Jul 23 23:57:58.288113 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.288096 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stvz9_47c58c88-b124-4e13-8596-79c1031522fa/routeoverride-cni/0.log" Jul 23 23:57:58.306620 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.306607 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stvz9_47c58c88-b124-4e13-8596-79c1031522fa/whereabouts-cni-bincopy/0.log" Jul 23 23:57:58.324371 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.324349 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stvz9_47c58c88-b124-4e13-8596-79c1031522fa/whereabouts-cni/0.log" Jul 23 23:57:58.659466 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.659396 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xdbwg_7933aa4d-a215-4a78-bcfd-ec12e3d6cf92/network-metrics-daemon/0.log" Jul 23 23:57:58.675738 ip-10-0-135-101 kubenswrapper[2568]: I0723 23:57:58.675715 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xdbwg_7933aa4d-a215-4a78-bcfd-ec12e3d6cf92/kube-rbac-proxy/0.log"